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I .   INTRODUCTION 

A.  PURPOSE 

The  purpose  of  this  thesis  is  to  analyze  the  integration 
of  Modeling  and  Simulation  (M&S)  into  the  U.S.  Army 
Operational  Test  and  Evaluation  (OT&E)  process.  The  elements, 
shortfalls,  and  recurring  problems  associated  with  the  OT&E 
system  are  examined  with  a  focus  on  those  that  can  be 
addressed  by  M&S.  From  this  analysis,  a  set  of 
recommendations  in  the  area  of  M&S  integration  into  Army  OT&E 
are  formulated  and  offered.  The  recommendations  address  both 
cost  effectiveness  and  adequacy  of  the  integrated  M&S  approach 
to  the  OT&E  process. 

B .  BACKGROUND 

The  modern  battlefield  requires  continued  advancement  in 
virtually  all  complex  weapon  systems.  Complex  systems  provide 
the  accuracy,  reduce  the  manpower  requirement,  and  create  the 
ability  to  control  vast  numbers  of  tasks  simultaneously.  They 
also  produce  a  challenge  to  the  OT&E  process  because  they  are 
extremely  difficult  to  test  in  an  operational  environment. 
This  difficulty  is  due  to  system  complexity  such  as  weapons 
tracking  systems  capable  of  identifying  and  tracking  multiple 
targets  or  communications  systems  capable  of  simultaneously 
processing  thousands  of  analog  and  digital  messages.   This 
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same  advancing  technology  has  also  accelerated  the  development 
and  useful  application  of  M&S  systems. 

Modeling  and  simulation  represents  an  explosive  growth 
industry  within  the  U.S.  Department  of  Defense  (DoD).  The 
technologies  comprising  these  revolutionary  techniques  are 
receiving  increasingly  wider  application  among  the  Services, 
and  in  particular  the  Army,  as  the  DoD  budget  continues  its 
rapid  decline.  As  the  cost  of  OT&E  continues  to  increase 
with  the  complexity  of  the  systems  evaluated,  the  application 
of  M&S  to  the  OT&E  process  becomes  ever  more  appealing. 
(Williams, 1993, p. 16) 

Counterbalancing  the  appeal  of  the  relatively  low  cost  M&S 
for  OT&E  applications  are  laws,  directives,  regulations  and 
policies  mandating  live  testing.  In  the  test  and  evaluation 
community,  there  is  a  widely  held  distrust  for  anything  that 
is  "simulated"  and  the  term  itself  brings  to  mind  a 
circumvention  of  generally  accepted  testing  protocols. 

The  connotation  of  any  form  of  M&S  within  the  test  and 
evaluation  communities  is  that  shortcomings  and  failures  of 
the  evaluated  system  are  being  masked  by  the  M&S  effort. 
Furthermore,  the  human  element  is  generally  thought  to  be 
eliminated  when  M&S  techniques  are  used.  The  following  quote 
from  Brigadier  General  Trifiletti,  Commanding  General  of  the 
U.S.  Army  Test  and  Experimentation  Command  (TEXCOM) 
illustrates  some  of  the  reasons  M&S  is  not  widely  accepted: 


The  fog  of  war  cannot  be  simulated  by  any  computer.  The 
effect  of  blisters  on  the  feet  of  your  soldiers,  as  well 
as  most  other  human  characteristics,  are  not  portrayed  in 
a  simulation.  You've  got  to  get  out  on  the  ground  with 
soldiers  and  equipment  to  understand  the  capabilities  and 
limitations.  (Trif iletti, 1994) 


C.  THESIS  OBJECTIVE 

This  thesis  analyzes  Army  OT&E  with  a  focus  on  M&S 
applications.  The  Army's  current  OT&E  process  is  analyzed  to 
identify  persistent  OT&E  problems  and  issues,  and  assess 
current  shortcomings  that  exist  in  OT&E.  Where  M&S  concepts 
have  potential  application  in  enhancing  OT&E  elements  or 
providing  solutions  for  OT&E  issues,  analysis  is  offered  as  to 
the  type,  extent,  and  applicability  of  M&S  concepts. 

Analysis  of  the  potential  strengths  and  weaknesses  of  M&S 
in  addressing  the  elements  and  issues  associated  with  the  OT&E 
process  are  presented.  Lessons  learned  from  past  OT&E  efforts 
are  analyzed  for  possible  process  improvement  through  M&S 
integration.  Supporting  conclusions  and  recommendations  based 
on  the  analysis  of  the  thesis  are  presented. 

D.  RESEARCH  QUESTIONS 

The  primary  research  question  of  this  thesis  is: 

•  To  what  extent  can  modeling  and  simulation  address  the 
recurring  problems  and  issues  associated  with  operational 
test  and  evaluation? 

The  four  subsidiary  research  questions  are: 

•  Does  the  operational  test  and  evaluation  process  lend 
itself  to  integration  with  modeling  and  simulation? 


What  current  and  proposed  models  and  simulations  are 
candidates  for  integration  with  operational  test  and 
evaluation? 

Can  operational  test  and  evaluation  costs  be  reduced 
through  the  use  of  modeling  and  simulation? 

Can  operational  test  and  evaluation  elements  be  enhanced 
through  the  use  of  modeling  and  simulation? 


E.   SCOPE  AND  LIMITATIONS 

This  thesis  focuses  on  M&S  applications  within  the  U.S. 
Army  OT&E  process.  The  potential  application  of  M&S  in  this 
thesis  is  limited  to  the  test  and  evaluation  phase  of  the 
acquisition  process.  Specifically,  this  thesis  addresses  OT&E 
rather  than  the  developmental  test  and  evaluation  (DT&E) 
process . 

This  thesis  identifies  potential  opportunities  to 
integrate  U.S.  Army  operational  test  and  evaluation  with 
current  and  future  M&S  applications  in  order  to  improve  the 
Army  OT&E  process.  The  laws,  regulations,  directives,  and 
policies  pertaining  to  OT&E  are  examined  to  identify  where  M&S 
integration  could  be  accomplished  and  under  what  circumstances 
that  M&S  is  prohibited.  Persistent  issues  surrounding  OT&E  on 
U.S.  Army  systems  identified  by  experienced  OT&E  test 
personnel,  are  examined  with  emphasis  on  applying  M&S  as 
methods  of  resolving  the  issues. 

Current  and  future  M&S  architecture  is  outlined  to  provide 
a  base  of  understanding  for  the  applicability  to  the  OT&E 


process  and  issues.   Army  wide  M&S  programs  and  applications 
across  numerous  commands  are  examined  and  presented. 

Analysis  of  M&S  integration  opportunities  includes  limited 
cost  analysis  and  other  value  added  properties  of  M&S. 
Networked  and  distributed  systems  are  emphasized  in  the  M&S 
structure. 

F.   RESEARCH  LITERATURE  AND  METHODOLOGY 

Research  data  were  obtained  from  official  Government 
directives  and  policies,  journals,  previous  theses,  United 
States  Code,  DoD  and  Army  regulations  and  manuals,  and 
personal  interviews.  Information  on  current  M&S  programs  was 
obtained  from  Army  Training  and  Doctrine  Command  (TRADOC) 
Analysis  Centers  (TRACs),  the  Army's  Simulation,  Training  and 
Instrumentation  Command  (STRICOM),  and  the  Defense  Modeling 
and  Simulation  Office  (DMSO).  Information  on  the  U.S.  Army 
operational  test  and  evaluation  process  was  obtained  from  the 
Operational  Test  and  Evaluation  Command  (OPTEC)  and  the  Test 
and  Experimentation  Command  (TEXCOM).  Current  directives  and 
policies  guiding  the  OT&E  process  were  reviewed. 

Research  was  conducted  via  personal  and  telephone 
interviews  with  cognizant  M&S  and  T&E  personnel.  Interviews 
with  OT&E  related  individuals  centered  around  the  OT&E 
processes  and  the  opportunities  for  M&S  to  enhance  OT&E 
elements  or  address  OT&E  problems  and  issues.  Interviews  with 


M&S  related  individuals  focused  on  the  state-of-the-art  in  M&S 
and  their  ability  to  address  the  OT&E  process. 

G.   ACRONYMS 

An  extensive  listing  of  acronyms  associated  with  both  OT&E 
and  M&S  subjects  is  presented  in  the  Appendix. 

H.   ORGANIZATION  OF  THESIS 

Chapter  II  of  this  thesis  addresses  the  OT&E  process 
including  OT&E  unigue  elements.  The  laws,  regulations, 
directives,  and  policies  shaping  the  OT&E  process  are  also 
presented  to  illustrate  M&S  opportunities  and  limitations  in 
Army  OT&E. 

Chapter  III  provides  an  overview  of  the  Army's  M&S  thrusts 
and  the  technologies  involved  in  the  M&S  efforts.  This 
chapter  addresses  both  current  and  immediate  future  M&S 
opportunities  that  may  have  application  in  the  OT&E  arena. 

Chapters  IV  and  V,  contain  the  analysis  of  M&S 
applications  in  the  OT&E  process  and  the  conclusions  and 
recommendations  offered,  respectively. 


II.   THE  OT&E  PROCESS 

A.   GENERAL 

The  most  capable  system  is  useless  if  it  cannot  be 
employed  by  the  intended  user  within  doctrine,  force 
structure,  tactics,  and  training  programs.  The  OT&E  process 
is  designed  to  assess  a  system's  effectiveness  in  a  realistic, 
operational  environment. 

The  purpose  of  this  chapter  is  to  delineate  Army  OT&E 
missions,  definitions,  and  organizations.  The  current  role  of 
M&S  in  the  OT&E  process  as  well  as  the  laws,  directives, 
regulations,  and  policies  guiding  the  Army  OT&E  process  are 
addressed. 

1.   OT&E  Missions  and  Definitions 

The  OT&E  missions  are:  To  determine  the  operational 
effectiveness  and  suitability  of  a  system  or  to  evaluate 
tactical  and  doctrinal  concepts,  under  realistic  combat 
conditions;  and  to  determine  if  the  minimum  acceptable 
operational  performance  requirements,  as  specified  in  the 
Operational  Requirements  Document  (ORD),  have  been  satisfied. 
(DoDI  5000.2, 1991, p. 8-5) 

Operational  effectiveness  and  suitability  refers  to 
the  weapon  system's  combat  effectiveness  achieved  when 
operated  and  maintained  by  typical  users,  within  the  intended 


doctrine  and  tactics,  and  as  part  of  standard  organizations 
integrated  with  other  battlefield  operating  systems. 
Operational  effectiveness  and  suitability  are  distinct  from 
weapon  system  capabilities  (e.g.,  speed,  range,  armament 
penetration,  etc.)  determined  during  DT&E. 

Realistic  combat  conditions  are  those  that  are 
representative  of  the  environment,  doctrine,  level  of 
training,  and  structure  in  which  the  evaluated  system  would 
normally  be  expected  to  operate  within  a  combat  environment. 
Realism  is  one  of  the  critical  elements  of  the  OT&E  process 
and  is  discussed  in  detail  later  in  this  chapter. 
2.   OT&E  Organization 

OT&E  is  organized  as  depicted  in  Figure  1.  Congress 
passed  legislation  creating  the  Office  of  Operational  Test  and 
Evaluation  in  1983  with  the  mission  to  evaluate  the  Services' 
weapon  systems  tests  and  assess  test  results.  As  part  of 
that  legislation,  the  Director  of  Operational  Test  and 
Evaluation  (DOT&E)  reports  directly  to  the  Secretary  of 
Defense  and  is  given  broad  authority  to  suspend  major  weapons 
programs  that  perform  poorly  in  operational  tests.  In 
addition  to  reporting  to  the  Secretary  of  Defense,  the  DOT&E 
reports  directly  to  both  the  House  and  the  Senate  Armed 
Services  Committees,  at  least  annually,  regarding  OT&E  of 
weapon  systems  being  developed.   The  goal  of  the  legislation 
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Figure  1  OT&E  Organization 


is  to  save  time  and  money  by  exposing  problems  before 
expensive  weapons  are  purchased  and  fielded. 

The  Operational  Test  and  Evaluation  Command  (OPTEC) 
was  established  15  November  1990  by  Secretary  of  the  Army 
General  Orders  Number  6.  It  consists  of  the  OPTEC 
Headquarters  and  Support  Agencies,  the  Operational  Evaluation 
Command  (OEC)  and  the  Test  and  Experimentation  Command 
(TEXCOM).  The  new  command  consolidates  previously  designated 
commands  and  agencies  including  the  TRADOC  Test  and 
Experimentation  Command  (TEXCOM),  the  former  Operational  Test 
and  Evaluation  Agency  (OTEA),  and  the  former  Acquisition  and 


Development  of  Threat  Simulators  Activity  (ADATS-A)  into  a 
single  command.  ADATS-A,  renamed  the  Operational  Threat 
Support  Activity  (OTSA),  and  the  Test  and  Evaluation 
Coordination  Offices  (TECOs),  are  incorporated  within  OPTEC 
Headquarters.  OPTEC 's  mission  is  to  conduct  and  monitor  user 
test  and  evaluation  (except  medical)  for  the  Army.  User  T&E 
includes  initial  and  follow-on  operational  test  and 
evaluation  ( IOTE  and  FOTE)  in  support  of  the  materiel 
acquisition  process,  force  development  testing  and 
experimentation  (FDTE),  concept  evaluation  program  (CEP) 
trials,  early  user  test  and  evaluation  (EUTE),  and  the  Army 
part  of  joint  test  and  evaluation  (JT&E). 

OPTEC,  Figure  2,  is  a  field  operating  agency  of  the 
Office  of  the  Chief  of  Staff  of  the  Army.  In  keeping  with  the 
Defense  Directives,  OPTEC  reports  the  results  of  Army  OT&E 
through  the  Vice  Chief  of  Staff  of  the  Army  directly  to  the 
Army  and  Defense  leadership.  The  main  part  of  OPTEC 's  mission 
is  the  planning,  conducting,  and  reporting  of  Army  OT&E  which 
has  been  required  by  law  since  1972.  Additionally,  OPTEC 
conducts  tests  for  TRADOC  in  support  of  its  mission  to  develop 
combat  doctrine,  organizations,  and  materiel  requirements. 
3.   OT&E  Support  to  the  Acquisition  Process 

The  materiel  acquisition  process  can  take  many  years 
from  the  time  a  materiel  requirement  is  identified  until  the 
system  is  fielded.   Although  in  this  process,  OT&E  accounts 
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Figure  2   OPTEC  Organizational  Structure 


for  only  a  short  time  period,  the  results  weigh  heavily  on  the 
decisions  to  continue  development,  accept  the  system,  or 
change  organization,  doctrine,  and  concepts. 

The  fundamental  purpose  of  OT&E  in  the  acquisition 
process  is  to  identify  the  areas  of  risk  associated  with  user 
requirements  and  acceptance  to  be  reduced  or  eliminated. 
During  the  early  phases  of  development,  T&E  is  conducted  to 
demonstrate  the  feasibility  of  conceptual  approaches,  minimize 
design  risk,  identify  design  alternatives,  and  estimate 
operational  effectiveness  and  suitability.  As  a  weapon  system 
progresses  through  the  developmental  process,  T&E  emphasis 
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turns  from  DT&E  towards  OT&E,  even  though  DT&E  and  OT&E  may 
not  be  conducted  sequentially.   (DSMC  TEMG, 1988,p. 1-1 ) 

After  a  weapon  system  has  successfully  completed  its 
development  phase,  there  is  tremendous  political  and 
bureaucratic  pressure  to  begin  full-scale  production.  The 
operational  testers  are  often  viewed  as  "show  Btoppers" 
because  successful  completion  of  OT&E  is  required  before  any 
developmental  system  can  progress  beyond  low  rate  initial 
production  (LRIP).  The  problem  is  compounded  by  the  current 
acquisition  process  which  schedules  critical  OT&E  events  just 
before  the  production  decision  milestone. 
(LeSueur, 1994, p. 12 ) 

Figure  3  depicts  Army  OT&E  events  in  each  acquisition 
phase.   The  IOTE  block  includes  CEP,  EUTE,  and  FDTE  events. 
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Figure  3  OT&E  Support  to  the  Acquisition  Process 


12 


During  the  Concept  Exploration  Phase  prior  to 
Milestone  I,  laboratory  testing,  modeling  and  simulations  are 
conducted  to  demonstrate  and  assess  the  capabilities  of  key 
subsystems  and  components.  Studies,  analyses,  simulation,  and 
test  data  are  used  to  explore  and  evaluate  alternative  concept 
designs  proposed  to  satisfy  user  generated  requirements.  OT&E 
conducted  during  this  phase,  called  early  operational 
assessment,  investigates  deficiencies  identified  during  the 
mission  area  analysis.  OPTEC  monitors  concept  exploration  T&E 
for  future  T&E  planning  and  to  provide  effectiveness  and 
suitability  inputs  desired  by  the  Program  Manager. 
Operational  assessments  addressing  the  operational  impact  of 
candidate  technical  approaches  are  conducted. 

During  the  Concept  Demonstration/Validation  phase, 
operational  effectiveness  and  operational  suitability 
assessments  are  conducted.  Information  on  tactics,  doctrine, 
personnel  requirements,  and  organization  impacts  of  the  weapon 
system  are  gathered.  OT&E  assessments  are  used  to  support  the 
Milestone  II  decisions  for  developing  promising  alternatives. 

The  objective  of  the  Engineering  and  Manufacturing 
Development  phase  is  to  design,  fabricate  and  test  systems 
that  closely  approximate  the  final  product.  Prior  to  the 
Milestone  III  decision,  a  dedicated  OT&E  is  conducted  on 
equipment  formally  certified  as  ready  for  "final  OT&E".  OT&E 
has  the  greatest  impact  on  major  programs  in  this  phase 
because  the  decision  to  proceed  beyond  LRIP  is  contingent  on 
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successfully  completing  the  IOTE.  A  formal  Operational 
Evaluation  (OPEVAL)  is  required  for  Milestone  III.  The 
purpose    of    OT&E    during    this    phase    is    to: 

•  Estimate  the  operational  effectiveness  and  suitability  of 
the  system. 

•Identify  operational  deficiencies. 

•  Recommend    and    evaluate    changes    in    production 
configuration . 

•  Provide  information  for  developing  and  refining  logistics 
support  requirements. 

•  Estimate   the   survivability   of   the   system   in   the 
operational  environment. 

Post-production  OT&E  are  Follow-on  Operational  Test 

and  Evaluation   (FOTE)   programs  designed  to  verify  the 

operational  effectiveness  and  suitability  of  the  production 

system  and  to  determine  if  deficiencies  identified  during  IOTE 

have  been  corrected.   FOTEs  also  refine  doctrine,  tactics, 

techniques,  and  training  programs  for  the  life  of  the  system. 

(DSMC  TEMG, 1988) 

B.   OT&E  LAWS,  REGULATIONS,  AND  DIRECTIVES 

The  OT&E  process  is  guided  by  numerous  laws,  regulations, 
directives,  and  policies.  The  provisions  of  these  mandates 
and  guiding  vehicles  directly  affect  how  the  OT&E  system 
plans,  conducts,  and  analyzes  evaluations.  The  role  of  M&S  in 
the  OT&E  process  is  limited  and,  in  some  cases,  precluded  by 
the  mandates  guiding  OT&E.  . 
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The  current  downsizing  of  the  Army  creates  pressure  to 

reduce  costs  at  every  level,   including  within  the  OT&E 

process.   Even  with  these  economic  pressures,  some  lawmakers 

are  continuing  to  guard  against  reductions  in  OT&E  testing. 

In  his  article  Fallout:    Weapons   that   don't   work?,    Rick  Maze 

reported  the  following: 

Senators  David  Pryor  (D-Ark)  and  William  Roth  (R-Del)  said 
they  oppose  a  Pentagon  procurement  reform  plan  that  would 
allow  operational  tests  to  be  waived  if  they  are  too 
expensive,  cause  long  delays  or  otherwise  interfere  with 
the  purchase  of  systems. 

Pryor  and  Roth  were  the  chief  sponsors  of  the  1983  law 
that  created  the  Pentagon's  Office  of  Testing  and 
Evaluation  and  remain  committed  to  the  principle  that 
weapons  need  to  be  fully  tested  in  live-fire  exercises 
before  large  numbers  are  built.  "It  is  a  very  important 
part  of  our  military  preparedness  to  make  sure  these 
weapons  work",  said  Pryor.  (Maze, 1994 , p. 20 ) 

Law  makers  and  policy  makers  have  reinforced  both  the 
requirement  for  OT&E  and  the  guiding  documents  addressed  in 
the  following  paragraphs. 

1.   Title  10,  United  States  Code 

United  States  Code,  Title  10,  Chapter  138  has  two 
sections  that  pertain  to  the  OT&E  process.  Section  2399, 
Operational  test  and  evaluation  of  defense  acquisition 
programs,  and  section  2366,  Major  systems  and  munitions 
programs:  survivability  and  lethality  testing;  operational 
testing. 

Section  2399  establishes  the  conditions  necessary  for 
a  major  program  (acquisition  category  I  and  II)  to  proceed 
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beyond  LRIP  and  the  DOTE  reporting  responsibilities.  This 
law's  provisions  state  that  a  major  defense  acquisition 
program  may  not  proceed  beyond  low  rate  initial  production 
(LRIP)  until  the  Initial  Operational  Test  and  Evaluation 
(IOTE)  for  that  program  is  complete.  It  further  directs  that 
the  DOTE  shall  analyze  the  results  of  major  systems'  OT&E  and 
prepare  a  report  for  the  House  and  Senate  Armed  Services 
Committees  and  the  Secretary  of  Defense.  The  report  shall 
state  the  DOTE ' s  opinion  regarding  OT&E  adequacy  and  whether 
the  results  indicate  that  the  evaluated  system  is  effective 
and  suitable  for  combat.   (USC, 1989, pp. 230-231 ) 

Section  2366  directs  the  Secretary  of  Defense  to 
ensure  that  no  major  system  proceed  beyond  LRIP  until  the  IOTE 
and  survivability/lethality  testing  is  complete.  This  section 
specifically  defines  the  conditions  required  for  survivability 
and  lethality  testing.  These  conditions  include  the 
requirement  for  live  firing  on  representative  systems  and 
therefore,  preclude  any  major  system  from  being  acquired 
solely  on  the  basis  of  M&S.  Contractor  and  other  personnel 
not  normally  involved  in  the  operation  or  support  of  a  weapon 
system,  are  forbidden  to  participate  in  OT&E  under  the 
provisions  of  this  section.   (USC, 1989 , pp. 581-582 ) 

2.   Department  of  Defense  Instruction  (DoDI)  5000.2 

This  instruction  references  and  reinforces  the 
provisions  of  Title  10,  United  States  Code,  outlined  in 
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paragraph  one,  above.  It  directs  that  OT&E  shall  be  designed 
to  support  the  decision  to  proceed  beyond  LRIP  and  establishes 
four  objectives  for  OT&E  testing: 

•  Provide   essential   information   for   assessment   of 
acquisition  risk  and  for  decisionmaking. 

•  Verify  attainment  of  technical  performance  specifications 
and  objectives. 

•Verify  that  systems  are  operationally  effective  and 
suitable  for  intended  use. 

•  Provide    essential    information    in    support    of 
decisionmaking. 

With  regard  to  the  use  of  M&S  in  the  OT&E  process, 
DoDI  5000.2  interprets  Title  10  in  a  strict  manner.  The 
Instruction  states  that  OT&E  does  not  include  an  operational 
assessment  based  exclusively  on  computer  modeling,  simulation, 
or  analyses  of  program  documents.  (DoDI  5000 . 2 , 1991, p. 8-2 ) 
3.   Army  Regulation  73-1,  Test  and  Evaluation  Policy 

Army  Regulation  73-1  (AR  73-1)  directs  the 
implementation  of  DoD  Directive  5000.1,  DoD  Instruction 
5000.2,  and  DoD  Manual  5000. 2-M.  This  regulation  describes 
the  type  of  OT&E  applications  to  be  provided  in  support  of 
each  phase  and  milestone.  It  links  T&E  to  the  acquisition 
process  and  directs  T&E  design  to  support  the  acquisition 
phases  and  milestones.  OPTEC  is  charged  with  the  overall 
management  of  Army  OT&E  programs  through  use  of  an  operational 
tester  (TEXCOM)  and  an  operational  evaluator  (OEC). 
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C.  CURRENT  ARMY  OT&E  AND  M&S  INTEGRATION 

Through  its  stated  policies,  OPTEC  recognizes  the  use  of 
M&S  to  achieve  adequate  realism,  support  economical  test 
execution,  and  provide  for  sufficiently  adequate  evaluations. 
However,  to  ensure  that  OPTEC  maintains  its  objective, 
independent  evaluator  perspective,  every  M&S  application  used 
in  the  OT&E  process  must  be  subjected  to  Verification, 
Validation,  and  Accreditation  (W&A)  before  any  data  or 
information  can  be  used.  M&S  applications  must  be  accredited 
for  each  specific  application,  and  the  W&A  conducted  for  one 
OT&E  does  not  apply  to  another.  Because  of  the  W&A 
requirement,  there  is  no  listing  or  catalog  of  M&S 
applications  available  for  use  in  the  OT&E  process.  (OPTEC 
73-21-1,1993) 

D.  OT&E  ELEMENTS 

The  OT&E  process  is  guided  by  operationally  related  test 
and  evaluation  elements.  These  elements  are  unique  to  the 
OT&E  process  and  drive  the  design  of  all  OT&E  programs. 

1.   Realism 

Realistic  environments  are  essential  to  achieving  the 
goals  of  OT&E.  A  realistic  environment  is  one  that  is 
representative  of  the  conditions,  doctrine,  level  of  training, 
and  structure  in  which  the  evaluated  system  would  normally  be 
expected  to  operate,  in  a  combat  environment.  These 
environments  are  distinctly  different  than  those  encountered 
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in  the  DT&E  phase  where  the  environment   is   carefully 
controlled. 

Realism  in  the  OT&E  test  process  is  affected  by  the 
resources  available  to  replicate  the  representative  threat  and 
friendly  force  array,  create  the  desired  combat  battlefield 
environment,  and  stress  the  evaluated  system  over  time.  Other 
factors  affecting  realism  include  the  degree  to  which  test 
participants  represent  typical  operational  personnel,  test 
personnel  familiarity  with  test  ranges  and  maneuver  areas,  and 
limitations  created  by  safety  or  environmental  concerns. 

The  conditions  present  during  the  OT&E  must  stress  the 
evaluated  system  within  the  doctrinal  envelope  of  operation. 
"Stimulators"  are  used  to  stress  communications  and  software 
intensive  components  of  the  evaluated  system  to  levels 
expected  in  a  combat  environment. 

An  OT&E  of  a  system  includes  its  interoperability 
characteristics.     A   realistic   structure   of   associated 
battlefield  operating  systems  is  essential. 
2 .   User  Oriented 

An  integral  part  of  the  OT&E  process  is  the  user. 
Representative  personnel  having  the  correct  organizational 
grade  and  specialty  structure,  level  of  training,  experience 
and  aptitudes  are  required  for  an  unbiased  operational 
evaluation  of  the  system. 
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Users  include  personnel  that  operate,  maintain, 
support,  or  provide  command  and  control  functions  affecting 
the  evaluated  system.  This  requires  that  the  interrelated 
battlefield  operating  systems  and  the  evaluated  system's 
support  structure  personnel  must  also  be  representative. 

The  term  "users"  specifically  excludes  contractor  and 
other  personnel  who  would  not  normally  be  involved  in  the 
operation,  control,  or  maintenance  of  the  evaluated  system. 
To  avoid  bias,  it  is  important  that  the  personnel  selected  to 
represent  the  user  are  indeed  representative. 
3.   Representative  Systems 

The  evaluated  system  equipment  must  be  sufficiently 
mature  to  be  considered  functionally  representative  of  the 
versions  eventually  fielded.  Software  must  be  complete  and  as 
near  to  the  fielded  version  as  possible  when  the  OT&E  is 
conducted.  All  associated  equipment,  publications,  training 
programs,  and  test/measurement /diagnostic  equipment  (TMDE) 
should  be  complete  prior  to  the  evaluation. 

The  evaluated  system  is  usually  part  of  a  larger  table 
of  organization  and  equipment  (TO&E)  and  interacts  or  is 
supported  by  other  standard  systems.  These  systems  must  also 
be  representative  of  those  found  in  an  operational 
environment. 


20 


4.   Sufficiency 

Sufficiency  in  the  OT&E  process  ensures  that  the  test 
plan  addresses  all  of  the  issues  specified  in  the  Operational 
Requirements  Document  (ORD),  the  data  collection  plan  provides 
for  sufficient  data,  and  the  evaluation  plan  is  sufficient  for 
the  system  evaluation.  The  structure  of  the  OT&E  vehicles  for 
acquiring  and  reducing  the  information  is  critical  for  a 
complete  and  unbiased  evaluation. 

The  test  plan  is  the  key  document  in  determining  what 
data,  and  under  what  circumstances  data  are  collected.  The 
test  plan  is  driven  by  the  user  provided  issues  specified  in 
the  ORD  and  the  requirements  for  data  supporting  the 
evaluation.  Sufficient,  unbiased  iterations  of  test  events 
must  be  accumulated  to  facilitate  the  evaluation  techniques 
prescribed  in  the  evaluation  plan. 

E.   SUMMARY 

The  OT&E  process  is  designed  to  assess  the  effectiveness 
and  suitability  of  systems,  concepts,  doctrine,  and  tactics  in 
a  realistic,  combat  environment.  The  results  of  OT&E 
assessments  have  significant  impact  on  the  acquisition  of 
major  programs  and  have  visibility  at  the  Congressional  level 
as  well  as  the  top  echelons  of  the  Army. 

The  use  of  M&S  in  the  OT&E  process  is  limited  by  law.  DoD 
and  Army  policies,  designed  to  protect  the  objectivity  of 
OT&E,  reflect  a  bias  against  the  use  of  M&S.  Counterbalancing 
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that  bias  are  budgetary  constraints  that  negatively  impact  on 
the  scope  of  "live"  testing. 

Advancing  technologies  have  vastly  improved  the 
capabilities  of  modern  M&S  applications.  The  next  chapter 
addresses  the  Army  M&S  thrusts  that  are  attempting  to  exploit 
these  advancing  technologies. 
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III.   ARMY  MODELING  AND  SIMULATION  THRUSTS 

A.   GENERAL 

From  the  Army's  point  of  view,  anything  short  of  actual 
combat  is  simulation.  Modeling  and  Simulation  (M&S) 
encompasses  applications  ranging  from  simple  mathematical  or 
physical  models  to  sophisticated  systems  integrating  three 
simulation  types;  "live",  "constructive",  and  "virtual".  The 
three  types  of  simulations  are  defined  below: 

1.  "live"  simulation  consists  of  operations  with  users 
employing  real  equipment  in  the  field 

2.  "constructive"  simulation  which  deals  with  wargames, 
models  and  analytical  tools 

3.  "virtual"  simulation  refers  to  systems  and  troops  in 
simulators  on  synthetic  battlefields  ( Singley, 1993, p. 35 ) . 

This  chapter  addresses  M&S  concepts  and  applications  that 
are  currently  available  or  are  being  planned  for  use  by  the 
Army  in  the  immediate  future.  The  M&S  applications  described 
are  not  the  only  resources  available,  but  are  representative 
of  the  technology  available  and  are  therefore  used  to 
illustrate  applications. 

The  Army's  M&S  resources  have  been  primarily  used  in  the 
areas  of  training,  testing,  material  development,  combat 
development,  and  analysis.  The  Army  has  increased  the 
capability  of  its  high  resolution  modeling  facilities  and  is 
making  progress  in  the  area  of  three  dimensional  simulation. 
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These  advancements  have  generated  high  expectations  that, 
within  the  next  few  years,  the  M&S  community  will  field  the 
requisite  fidelity  and  distributed  capabilities  needed  to 
streamline  the  current  acquisition  process.  (Crouch, 1994, p. 3) 

Technological  components  of  the  Army  simulation  capability 
include  a  Distributed  Interactive  Simulation  (DIS)  environment 
transmitted  over  the  Defense  Simulation  Internet 
communications  network.  DIS  provides  protocols  that  enable 
the  communication  between  various  models  such  as  Janus  (a  high 
resolution  constructive  model),  semi-automated  forces  (SAFOR) 
generator,  and  the  Battlefield  Distributed  Simulation  - 
Developmental  (BDS-D)  virtual  simulator  complex. 

The  highest  echelons  of  Army  leadership  recognize  the  need 

to  exploit  advanced  M&S  capabilities  for  weapon  system 

acquisition  in  the  post-cold  war  era.   In  the  "United  States 

Army  Posture  Statement   FY95" ,  The  Honorable  Togo  D.  West,  Jr., 

Secretary  of  the  Army,  and  Army  Chief  of  Staff  General  Gordon 

R.  Sullivan  state: 

The  Army  will  maintain  technological  superiority  through 
pursuit  of  promising  advanced  technologies  and  concepts, 
developing  new  systems  when  existing  systems  have  reached 
the  end  of  their  useful  lives  or  when  a  new  system  offers 
an  essential,  revolutionary  combat  capability.  We  will 
exploit  Advanced  Distributed  Simulation  for  better,  more 
affordable  requirements  and  acquisition  testing  and  will 
reduce  acquisition  costs  by  reducing  infrastructure  and 
development  cycle  times.   (West, 1994, p. 87 ) 

They  also  outline  two  major  assets  the  Army  will  use  to 

develop  weapon  systems;  the  Louisiana  Maneuvers  (LAM)  and  the 

Battle  Labs.   Both  of  these  assets  integrate  M&S  into  the 
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materiel  and  concept  development  process  and  will  be  discussed 
in  detail  later  in  this  chapter. 

B.   DoD  M&S  POLICIES 

The  Army  is  adopting  policies  to  take  advantage  of 
advanced  M&S  opportunities.  The  Army's  leaders  are  setting 
the  M&S  policy  cornerstones  as  evidenced  by  Army  Chief  of 
Staff,  General  Gordon  Sullivan's  statement: 

You  need  to  know  that  we  will  use  simulation  techniques 
throughout  the  Army's  acquisition  process.  We  will 
determine  needs  in  large-scale,  simulation-supported 
exercises  that  allow  us  to  consider  alternative  solutions 
that  meet  our  needs.  We  will  use  drawings,  diagrams  and 
3-dimensional  models  generated  by  computers,  put  them  in 
constructive  or  virtual  environments,  and  compare 
alternatives  both  technically  and  tactically.  The  most 
promising  technologies  will  be  tested  by  real  soldiers, 
first  in  reconf igurable  crew  stations,  then  in  full  scale 
simulations.  Final  designs,  production  and  assembly  steps 
will  also  be  simulated  in  virtual  factories  before  actual 
prototypes  are  made.  Then  the  actual  and  virtual 
prototypes  will  be  exercised  simultaneously  to  discover 
potential  problems  before  production  begins.  Tactics, 
techniques,  and  procedures  are  also  developed  along  with 
the  system  so  that  the  system  is  fully  ready  for  use  when 
produced.  (Sullivan, 1993 ) 

Lieutenant  General  William  H.  Forster,  the  Military  Deputy 

to  the  Assistant  Secretary  of  the  Army  (Research,  Development 

and  Acquisition),  further  refined  the  M&S  focus  in  his  24  May 

1993  Memorandum  for  the  Deputy  Commanding  General,  U.S.  Army 

Materiel  Command  and  all  Program  Executive  Officers.   Under 

the  subject  of  "Simulation  Support   to  the  Army",    LTG  Forster 

stated: 
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The  Army  Science  Board  and  Defense  Science  Board  have 
recently  studied  the  potential  improvements  to  DoD 
acquisition  offered  by  advanced  simulation,  particularly 
Distributed  Interactive  Simulation  (DIS).  Both  concluded 
that  simulation  can  improve  acquisition  from  concept  to 
fielding  through  such  innovations  as:  virtual 
prototyping;  engineering  simulation;  linking  of 
constructive,  virtual  and/or  live  simulations;  assisting 
the  user  in  execution  of  experiments  in  employment 
tactics;  user  test  design  and  critical  issue 
identification;  and  improved  training  prior  to  fielding. 

The  Army  is  leading  the  way  for  DoD  in  simulation  with 
such  initiatives  as  Battlefield  Distributed  Simulation- 
Developmental,  Close  Combat  Tactical  Trainer,  and  DIS 
Modernization  and  Master  Plans.  We  need  to  take  full 
advantage  from  concept  to  fielding.  Effective  second 
quarter  fiscal  year  1994,  all  Army  acquisition  strategies 
for  Acquisition  Category  I  and  II  programs  will  include  a 
simulation  support  plan.  Additionally,  the  simulation 
support  plan  must  be  included  in  the  Program  Manager's 
ASARC  briefing.  Other  programs  may  be  tasked  by  the  Army 
Acquisition  Executive  to  include  a  simulation  support 
plan.  (Forster, 1993, p. 1) 

Policies  integrating  M&S  into  the  acquisition  process  have 

been  initiated  from  the  top  levels  of  the  Army.   The  Battle 

Labs  and  LAM  have  been  established  and  charged  with  pursuing 

new   technologies   and   investigating   concepts   using   an 

integrated  M&S  approach.   Policies  for  integrating  M&S  into 

the  OT&E  portion  of  the  acquisition  process  have  begun,  but 

not  fulfilled. 

C.   M&S  CONCEPTS  AND  APPLICATIONS 

The  Army  plans  to  link  M&S  technological  components 
together  through  networks  such  as  the  Defense  Systems  Internet 
(DSI)  to  create  environments  like  the  Distributed  Interactive 
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Simulation  (DIS)  system.   These  M&S  component  concepts  and 
assets  are  discussed  in  detail  in  the  following  sections. 
1 .   Model-Test-Model 

Model-Test-Model  (MTM)  is  a  three  phase  process 
designed  to  assist  DoD  and  the  Army  in  conducting  more 
effective  and  efficient  test  and  evaluation  of  new  weapon 
systems.  The  concept  envisions  the  synergistic  interaction  of 
testing  and  modeling  to  enhance  both  operational  testing  and 
combined  arms  modeling.  The  first  phase  examines  the  test 
design  to  identify  possible  efficiencies  and  effectiveness 
improvements  of  the  design.  The  second  phase  examines  the 
conduct  of  the  test  to  gain  insights  into  potential  model-test 
differences.  The  final  phase  focuses  on  the  amount  of 
correlation  or  association  between  the  model  and  test  results. 
Based  on  the  amount  of  correlation,  the  model  may  be 
accredited  for  a  specific  trial.  Once  trials  or  missions  are 
accredited,  test  results  may  be  extended  beyond  test  scenarios 
and  conditions  which  may  be  cost  prohibitive  or  constrained 
for  environmental  or  safety  reasons.   ( IR  MTM, 1994) 

A  basic  building  block  of  MTM  is  a  model,  which  is 

defined  as: 

A  representation  of  an  actual  or  conceptual  system  that 
involves  mathematics,  logical  expressions,  or  computer 
simulations  that  can  be  used  to  predict  how  the  system 
might  perform  or  survive  under  various  conditions  or  in  a 
range  of  hostile  environments.   (DSMC  TEMG, 1988, p. B-8) 
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2.   Janus  Combat  Model 

Janus  is  a  computer-based,  two-sided  combat  simulation 
model.  Janus  UNIX  3.17,  and  Janus  Virtual  Memory  System  (VMS) 
4.0,  are  the  most  recent  versions  of  the  model  that  are  now  in 
common  use.  (Pate, 1992 , p. 2  ) 

The  Janus  combat  model  accommodates  up  to  600 
individual  combat  systems,  including  up  to  100  indirect  fire 
systems,  for  both  threat  and  friendly  forces.  All  systems  are 
capable  of  moving,  detecting,  and  firing  over  a  50  square 
kilometer,  three  dimensional  terrain  representation.  Combat 
systems  ranging  from  major  individual  fighting  platforms  such 
as  tanks  and  helicopters  to  dismounted  infantry,  are  portrayed 
using  the  attributes  of  the  real  or  notional  systems  being 
modeled  (e.g.,  size,  speed,  sensors,  armament,  ballistic 
protection,  thermal/optical  contrast,  etc.).  The 
vulnerability  of  each  system  is  characterized  by  data  sets  of 
probability  of  hit  (PH)  and  probability  of  kill  (PK)  that 
individually  associate  each  combat  system  with  each  weapon  in 
the  simulation.  (Crooks, 1992 ) 

Janus,  because  of  its  high  resolution  capabilities, 
has  been  primarily  used  in  aiding  analysts  with  Cost  and 
Operational  Effectiveness  Analysis  (COEA)  and  combat 
development  system  studies  such  as  the  9th  Infantry  Division 
(Motorized)  force  design.  Recently,  the  concept  of 
integrating  a  constructive  model  (Janus)  with  a  virtual 
simulation  environment  was  explored.   One  of  the  Anti-Armor 
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Advanced  Technology  Demonstration  thrusts  is  to  merge 
constructive  and  virtual  simulation  worlds.  This  allows  the 
orchestration  of  selected  forces,  within  the  virtual 
simulation  environment,  to  be  controlled  from  a  constructive 
model  such  as  Janus.  This  orchestration  i6  referred  to  as 
semi-automated  forces.  (Crouch, 1994 ,pp. 49-50) 
3.   Semi-Automated  Forces 

The  semi-automated  forces  (SAFOR)  capability  allows  a 
single  individual  operating  a  constructive  model  to  control 
various  sized  units  such  as  platoons,  companies,  or  battalions 
within  a  virtual  simulation.  These  forces  appear  on  the 
virtual  battlefield  just  as  manned  simulators  do;  the  fact 
that  they  are  SAFOR  is  transparent  to  the  other  participants. 
To  keep  costs  associated  with  experiments  within  budgetary 
constraints,  SAFOR  can  be  used  to  represent  both  friendly 
(adjacent,  supporting,  higher  echelon  and  lower  echelon 
elements)  and  threat  forces.    (Loral, 1992 , p. 2 ) 

SAFOR  is  useful  from  both  a  command  and  control  and  a 
cost  savings  perspective.  Command  and  control  is  enhanced  by 
the  capability  to  control  several  systems  on  the  virtual 
battlefield  from  a  constructive  model.  Savings  are  derived 
from  constructive  models  that  generate  a  system  on  the  virtual 
battlefield  at  a  lower  cost  than  those  generated  by  virtual 
networks  such  as  BDS-D.   (Crouch, 1994, p. 51 ) 
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4.   Battlefield  Distributed  Simulation  -  Developmental 

Battlefield  Distributed  Simulation  -  Developmental 
(BDS-D)  provides  the  technological  framework  to  conduct  a 
simulated  battlefield  over  a  distributed  network.  Through 
this  system,  individuals  can  fight  and  analyze  the 
effectiveness  differences  resulting  from  the  changes  made  in 
equipment,  doctrine,  tactics,  organizations,  and  training 
methods.  The  sequence  of  the  battle  can  be  recorded  and  later 
analyzed  in  detail  to  refine  those  changes. 

The  supporting  technologies  create  a  simulated  or 
"virtual"  battlefield  on  which  users  can  conduct  cost 
effective  experiments  or  training  exercises.  The  exercises 
are  conducted  using  actual  soldiers  operating  simulators, 
permitting  soldier-in-the-loop  experimentation.  Through  a 
combination  of  local  area  and  extended  distance  networks, 
soldiers  operating  simulators  at  one  site  are  able  to  see  and 
interact  with  soldiers  operating  at  other  sites  on  a  common 
digitized  battlefield.  (Loral , 1992 , p. 3 ) 

The  BDS-D  program  is  sponsored  by  the  U.S.  Army  and 
the  Commander,  U.S.  Army  Simulation,  Training,  and 
Instrumentation  Command  (STRICOM)  is  the  Program  Manager  for 
the  BDS-D  effort.  STRICOM  provides  the  focal  point  between 
DoD  agencies,  user  agencies,  industry,  and  the  BDS-D  sites. 
(Loral, 1992, p. 4) 

BDS-D  will  support  experiments  and  evaluations  in  a 
variety  of  areas.  Using  the  approach  of  simulating  before  and 
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during  procurement  of  a  new-  weapon  system,  users  are  able  to 
experiment  with  the  weapon  system  design  throughout  its 
acquisition  life  cycle.  For  example,  developers  can  perform 
the  following: 

•  Define  requirements  accurately  and  assess  trade-offs. 

•  Explore  the  capabilities  that  should  be  incorporated  into 
a  new  or  existing  system. 

•  Investigate  the  density  and  allocation  of  the  system  that 
achieves  optimum  performance  on  the  battlefield. 

•  Determine  the  best  means  to  employ  the  system  once  it  is 
built  (Loral, 1992, p. 4) . 

Users  can  experiment  with  new  and  innovative  ways  of 
employing  weapon  systems  so  that  they  better  realize  their 
design  potentials.  Changes  in  organizational  structure  can 
also  be  analyzed  to  determine  the  relative  effectiveness  on 
the  battlefield  of  competing  organizations.  (Loral, 1992, p. 4 ) 

Local  area  networks  consisting  of  low  cost  battlefield 
simulators,  and  simulations  of  experimental  systems,  and  SAFOR 
are  required  to  facilitate  this  effort.  These  simulators  will 
be  linked  together  through  a  network  such  as  DSI  to  provide 
virtual  combat  operations  in  the  DIS  environment  for  materiel 
and  combat  development,  and  operational  testing  exercises. 
(Kelly, 1993, p. 19) 

5.   Distributed  Interactive  Simulation 

One  of  the  challenges  is  integrating  constructive  and 
virtual  simulation  technologies.    The  dissimilar  computer 
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systems  and  simulators  that  the  Army  intends  to  combine  into 
one  integrated  network  were  not  designed  to  communicate  with 
one  another.  To  achieve  this  desired  seamless  simulation 
capability,  a  simulated  environment  that  provides  standard 
terms  and  protocols  which  allows  these  different  computer 
systems  and  simulators  to  communicate  is  required.  The 
standardization  of  terms  is  achieved  by  Distributed 
Interactive  Simulation  (DIS)  Protocol  Data  Units  (PDUs). 
(Crouch, 1994, p. 56) 

DIS  creates  a  synthetic  environment  within  which 
humans  may  interact  through  simulation  at  multiple,  networked 
sites  using  compliant  architecture,  modeling,  protocols, 
standards,  and  data  bases.  DIS  and  its  PDUs  are  the  next 
generation  of  distributed  simulation  evolving  from  the 
Advanced  Research  Project  Agency's  (ARPA)  research  project  of 
the  1980 's  known  as  Simulation  Network  (SIMNET).  The  DIS  is 
in  its  initial  stages  with  many  obstacles  yet  to  overcome. 
(1ST, 1993, p. 4) 

DIS  will  take  advantage  of  currently  installed  and 
future  simulations  manufactured  by  different  organizations. 
Consequently,  interoperability  between  dissimilar  simulations 
is  absolutely  required.  The  first  step  in  achieving  this 
interoperability  is  to  develop  a  communications  protocol. 
There  must  be  a  standard  set  of  messages  that  communicate 
between  host  computers,  the  states  of  simulated  and  real 
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entities,   and  their  interactions.    This  information  is 
communicated  through  DIS  PDUs .  ( 1ST, 1993, p. 8 ) 
6.   Defense  Simulation  Internet 

DIS  operations  are  supported  by  a  communication  system 
known  as  the  Defense  Simulation  Internet  (DSI).  This 
communication  system  was  developed,  and  is  currently  operated 
by  Advanced  Research  Projects  Agency.  The  DSI  consists  of 
commercial  telephone  circuits  over  existing  networks  with 
nodes  at  user  locations,  and  strategically  placed  switching 
nodes  (Fix  East  and  Fix  West)  with  a  central  controlling 
facility  in  Chicago,  Illinois.  Connectivity  is  made  with 
military  and  civilian  satellites  to  allow  worldwide, 
simultaneous  DIS  operations.  (DIS  MODPLAN, 1993, p. 2 ) 

There  are  approximately  30  DSI  nodes  supporting  all 
Services'  command  posts,  Battle  Simulation  Centers,  test  beds, 
Battle  Labs,  research  centers,  unified  commands,  and  civilian 
contractors  that  support  the  military.  The  DSI  is  expected  to 
expand  over  the  next  year  with  approximately  25  additional 
sites.  (DIS  MODPLAN, 1993, p. 2) 

Each  location  connected  to  the  DSI  network  is  referred 
to  as  a  DSI  node.  The  Army  operates  two  TRADOC  Battle  Lab 
nodes  on  the  DSI;  Fort  Knox  and  Fort  Rucker  with  Fort  Rucker 
designated  as  a  Battle  Lab  support  facility.  Eventually  the 
Army  wishes  to  operate  six  to  eight  Army  Battle  Lab  nodes  and 
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additional   communications  nodes  at  most  major  commands 
throughout  the  Army.  ( Singley, 1993, p. 37 ) 

Through  the  interpretive  capabilities  of  DIS  and  the 
communication  network  established  by  the  DSI,  the  Army  will 
attempt  to  link  constructive  models  such  as  Janus  with  virtual 
simulation  networks  such  as  BDS-D.   (Crouch, 1994, p. 57 ) 

D.   BATTLE  LABS 

TRADOC  has  organized  six  Battle  Labs  to  identify,  develop, 
and  experiment  with  new  warfighting  concepts  and  new 
capabilities  offered  by  emerging  technologies.  The  Battle 
labs  are  distributed  as  depicted  in  Figure  4. 


BATTLE  LABS 


BATTLE  COMMAND 
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EARLY  ENTRY 
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BATTLE ■ SPACE 
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BATTLE -SPACE 
DISMOUNTED 
FT  BENNING,  GA 


Figure  4  Battle  Labs 
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The  Battle  Labs  initiative  is  a  response  to  the 
unpredictability  of  the  post-cold  war  world.  The  wide  variety 
and  rapidly  changing  array  of  direct  and  indirect  threat 
situations  have  replaced  the  single,  well-defined  threat  of 
the  Warsaw  Pact  which  drove  doctrine  and  materiel  requirements 
during  the  cold-war  era.  (BLG, 1993, p. 3) 

Battle  Labs  maintain  the  Army  perspective  across  a  wide 
spectrum  of  ideas  and  concepts  to  ensure  that  the  Army  remains 
dominant  on  future  battlefields.  Through  conceptualizing, 
analyzing,  simulating,  testing,  and  evaluating  projects, 
Battle  Labs  aggressively  seek  out  emerging  concepts  and 
technologies  worldwide.  They  streamline  the  Army  planning 
process  by  providing  an  organized  way  to  define  requirements; 
allowing  industry  to  develop  a  focus  for  developmental  work; 
providing  industry  access  to  a  pool  of  Army  thinkers  who  can 
delineate  ideas  about  modernization  alternatives;  and  linking 
combat  lessons  learned,  exercise  results,  and  insights  from 
the  regional  Commanders  in  Chief  to  a  formal  analytical 
testbed.   (West, 1994, p. 78 ) 

Battle  Labs  will  be  linked  through  the  DSI  to  each  other, 
the  Army  R&D  Community,  sister  Services,  DoD,  and  national 
agencies.  All  are  organizing  to  take  advantage  of  the  DIS 
technology,  which  was  not  available  to  earlier  te6t-bed 
operations.  BDS-D  networks  will  allow  warfighting  experts  at 
TRADOC  Centers  to  advance  ideas  and  test  them  by  simulation  at 
a  number  of  locations.  (BLG, 1993, p. 4 ) 
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Battle  Labs  will  provide  the  tools  and  standards  to 
simulate  activities,  at  a  high  level  of  realism,  from  theaters 
of  war  to  factories  and  manufacturing  processes.  The 
mechanism  for  entry  into  the  synthetic  battlefields  has  been 
limited  to  a  few  networked  simulators  and  individual 
workstations.  These  will  be  greatly  expanded  in  the  1990s  to 
include  the  reconf igurable  BDS-D  simulators  that  will  provide 
the  desired  mix  of  real  ranges,  virtual  simulations  and 
aggregated  constructive  simulations  into  wargame 
representations.  Multipurpose  surrogates,  such  as  SAFOR, 
supported  by  computer  emulation,  will  allow  soldiers  to 
participate  or  to  be  simulated  in  battles.  (BLG, 1993, p. 6 ) 

Simulation  tools  and  methodologies  integrated  into  Battle 
Labs  offer  industry  a  new  area  of  innovative  development  that 
has  potential  beyond  its  military  application.  The  use  of 
concurrent  engineering  principles  reduces  development  time 
and  speeds  the  acquisition  process.  Virtual  prototypes  will 
be  produced,  so  design  and  manufacturing  tradeoffs  can  be 
evaluated.  Eventually  the  manufacturing  process,  the  military 
system,  and  the  system's  performance  may  all  be  modeled  and 
refined  before  the  first  piece  of  hardware  is  built. 
(BLG, 1993, p. 6) 

Synthetic  environments  will  not  completely  replace 
hardware  demonstrations  as  a  means  of  introducing  new 
capabilities  to  the  user.  However,  given  the  increased  costs 
of   hardware   development   and  test,   contrasted  with  the 
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decreased  costs  and  increasing  fidelity  of  reconf igurable 
simulators,  synthetic  environments  become  ever  more  appealing. 
(BLG,1993,p.7) 

Battle  Labs  will  help  prepare  the  Army  for  the  challenges 
of  the  next  century.  Unlike  the  manpower-intensive  Louisiana 
Maneuvers  of  the  1940s,  simulation  and  Battle  Labs  will  afford 
a  basis  for  the  Louisiana  Maneuvers  of  the  1990s  and  beyond. 
(Ross, 1993, p. 18) 

E.   LOUISIANA  MANEUVERS 

The  term  Louisiana  Maneuvers  (LAM)  refers  to  a  series  of 
large-scale  military  exercises  carried  out  by  Army  General 
George  C.  Marshall  in  Louisiana  in  1941  to  rebuild  and  renew 
the  Army  for  combat  in  World  War  II.  The  maneuvers  met  the 
challenge  by  training  soldiers,  units,  and  headquarters; 
validating  and  improving  doctrine  and  equipment;  and  testing 
new  units  and  concepts. 

The  Army  reestablished  the  LAM  in  1992  under  a  Chief  of 

Staff  of  the  Army  charter: 

The  Louisiana  Maneuvers  will  energize  and  focus  the  Army 
on  warfighting  and  its  Title  X  responsibilities,  will 
provide  the  Army's  senior  leadership  strategic  agility  in 
decision  making,  and  will  assess  the  Army's  direction  and 
progress  on  its  journey  into  the  21st  Century. 

LAM  allows  the  Army  to  think,  to  grow,  and  to  effectively 
take  charge  of  the  process  of  change.  It  provides  the 
Army  leadership  a  systematic  approach  to  explore  and 
examine  doctrine,  organization,  training,  materiel,  leader 
development,  and  soldier  issues  shaping  the  force  for  the 
next  century,  without  putting  thousands  of  soldiers  in  the 
field.  By  harnessing  the  power  of  the  microprocessor,  the 
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Army  will  use  simulations  to  develop  and  produce  new 
equipment,  to  enhance  combat  readiness,  to  train,  and  to 
experiment  with  new  ideas.  LAM  uses  exercises,  battle 
laboratories,  and  other  mechanisms  to  cover  the  full  range 
of  military  operations.   (West, 1994, pp. 76-77 ) 

Some  major  issues  that  LAM  will  examine  in  the  near  future 
include  an  effectiveness  review  of  command,  control, 
communications,  computers,  and  intelligence  (C4I) 
architectures;  implications  of  around-the-clock  operations; 
the  impact  of  new  technologies  and  equipment  on  the 
battlefield;  deployability,  lethality,  and  survivability  of 
light  and  heavy  forces;  the  impact  of  weapons  of  mass 
destruction  in  today's  security  environment;  and  a  broad  range 
of  force  generation  and  deployment  requirements. 
(West, 1994, p. 77) 

Relying  heavily  upon  simulation  and  modeling  technologies, 
senior  Army  leaders  will  use  LAM  as  a  tool  to  help  save  money 
by  speeding  the  introduction  of  promising  new  weapon  systems. 
This  is  derived  from  quickly  eliminating  unworkable  concepts, 
aiding  in  the  development  of  new  doctrine,  and  generally 
guiding  the  Army  as  it  reshapes  itself  for  meeting  post-cold 
war  missions.  (Holzer , 1993, p. 36) 

F.   SUMMARY 

The  Army  is  moving  to  take  advantage  of  advanced  M&S 
opportunities  and  the  potential  cost  savings  M&S  offers  in  a 
fiscally  constrained  post-cold  war  era.  The  Army's  leadership 
is  setting  policies  that  endorse  M&S  proliferation  Army  wide, 
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including  in  the  costly  acquisition  process.  As  a  vital  part 
of  the  acquisition  process,  how  can  OT&E  integrate  M&S 
concepts  and  applications  to  address  recurring  problems  and 
shortcomings,  and  enhance  the  OT&E  process? 
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IV.   ANALYSIS 

A.*  GENERAL 

The  OT&E  community  has  a  most  difficult  role  in  defining 
when,  where  and  how  it  can  participate  outside  the  live 
environment.  Operational  testers  find  themselves  caught 
between  United  States  Code,  Title  10  mandates  and  the  spiral 
of  diminishing  defense  dollars.  This  is  a  most  dangerous 
position.   (Crouch, 1994, p. 86) 

The  "most  dangerous  position"  refers  to  a  situation  where 
the  OT&E  independent  evaluator  (OPTEC)  does  not  integrate  M&S 
and  continues  to  use  expensive  live  testing,  while  materiel 
and  combat  developers  push  for  OT&E  certification  based  on 
their  own,  cost  effective  operational  M&S  applications. 
Although  undesirable,  the  loss  of  the  independent  OT&E  view 
could  result  from  the  continuing  economic  pressures  in  the 
post-cold  war  era. 

The  Defense  Modeling  and  Simulation  Office  (DMSO)  is 
funding  simulation  efforts  to  test  the  merits  of  using  other- 
than-live  environments  for  operational  testing.  The  customers 
for  this  effort  are  primarily  the  materiel  and  combat 
development  communities.  Data  are  still  insufficient  to 
assess  the  current  ability  to  utilize  other-than-live 
environments  for  operational  testing.   However,  technology 
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advancements  in  the  immediate  future  are  expected  to  provide 
the  required  simulation  fidelity  that  will  make  virtual 
simulation  a  viable  alternative  for  some  early  and  follow-on 
portions  of  operational  assessments.   (Crouch, 1994, p. 87 ) 

This  chapter  examines  OT&E  issues  and  problems,  and 
analyzes  M&S  capabilities  to  resolve  them  or  reduce  their 
severity.  The  methodology  used  and  OT&E  problem  and  issue 
analysis  are  addressed  in  the  following  sections. 

B .   METHODOLOGY 

The  major  OT&E  problems  and  issues  were  established  by 
conducting  interviews  with  experienced  OT&E  professionals 
assigned  to  the  TEXCOM  Experimentation  Center  (TEC)  located  at 
Fort  Hunter-Liggett ,  California.  The  OT&E  problems  and  issues 
were  identified  from  OT&Es  planned  or  conducted  at  TEC,  Fort 
Hunter-Liggett . 

1.   OT&E  Weapon  Systems 

OT&E  tests  were  planned  and  conducted  on  the  six 
weapon  systems  briefly  described  below: 

•  M1A2  Abrams  Main  Battle  Tank.  Upgrades  to  the  Ml  Abrams 
include  improved  computer  and  armor  protection  systems. 
This  weapon  system  is  designed  to  be  fielded  in  armored, 
mechanized  infantry,  and  cavalry  units,  ranging  from 
platoon  to  division  sized  elements. 

•  Air-to-Air  Target  Designator.  This  device  uses  laser 
technology  to  designate  airborne  targets  for  armed  scout 
and  attack  helicopters  assigned  to  air  attack  and  air 
cavalry  units. 

•  Multi-Spectral  Combat  Decoy.  This  system  provides  visual 
and  thermal  tracked  vehicle  signatures  in  a  lightweight 
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kit  form  for  use  by  armored,  mechanized  infantry,  and 
counter-intelligence  units. 

•Mini  Eyesafe  Laser  Infrared  Observation  System  (MELIOS). 
MELIOS  was  designed  to  improve  the  nignt  observation 
capabilities  for  combat,  combat  support,  and  combat 
service  support  units  at  every  echelon  from  platoon 
through  corps  level. 

•  Javelin  Anti-Tank  Missile  System.  The  Javelin  is 
replacing  the  shoulder-fired  Dragon  missile  system  and 
provides  extended  range  and  improved  attack  attitudes.  It 
will  replace  all  Dragon  systems  in  combat  and  combat 
support  units  from  platoon  through  corps  level. 

•  AH-64D  Longbow  Apache  Helicopter.  Upgrades  to  the  AH-64D 
Apache  include  improved  armament  (including  Longbow 
Hellfire  missiles),  automated  systems,  engines,  and  fire 
control  radar.  Longbow  Apaches  will  replace  attack 
helicopters  currently  fielded  in  attack  helicopter  units 
at  battalion  through  corps  echelons. 

Interviews  were  also  conducted  with  personnel  from 

OPTEC,  TRAC  Monterey,  and  TRAC  White  Sands  Missile  Range 

(WSMR),  New  Mexico.   These  interviews  identified  the  level  of 

M&S  integration  for  the  OT&E  programs  examined. 

2.   OT&E  Problem  and  Issue  Categories 

The  problems   and   issues   are  divided   into   five 

categories.    The  five  OT&E  problem  and  issue  categories 

include  Test  Design  Validation,  Resource  Constraints,  System 

Component  Stress,  Safety  and  Environmental  Concerns,  and  Data 

Reliability  and  Validity.   Categories  were  selected  when  at 

least  two  of  the  six  OT&E  programs  experienced  similar 

problems  and  issues.   The  categories  and  general  types  of 

problems  and  issues  assigned  to  each  are  described  in  the 

following  paragraphs. 
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a.    OT&E  Test  Design  Validation  Category 

Test  issues  and  problems  were  categorized  under 
OT&E  Test  Design  Validation  when  shortcomings  were  associated 
wfth  test  design  errors  or  omissions.  The  validation  process 
is  designed  to  ensure  that  the  baseline  scenarios  and  test 
execution  plans  satisfy  the  user  and  evaluator  requirements 
for  data  collection,  while  simultaneously  meeting  the  mandates 
and  guidance  for  the  conduct  of  OT&Es. 

A  desired  method  for  test  design  validation  is  the 
use  of  a  pilot  test.  A  pilot  test  is  a  rehearsal  of  test 
events  using  test  participants,  fully  instrumented  candidate 
systems,  data  collection  resources,  and  test  ranges  and 
maneuver  areas,  but  excluding  data  collected  from  the  test 
report.  A  pilot  test  is  useful  in  determining  if  the  data 
collected  address  the  criterion  under  consideration  for 
specific  events.  It  usually  does  not  determine  whether  or  not 
the  test  design  fully  addresses  all  critical  issues 
identified.  The  pilot  test  advantages  include  validation  of 
test  scenarios,  data  collection  plans,  test  facilities,  and 
pre-test  training  with  no  test  report  impact.  Disadvantages 
include  time  and  funding  requirements,  and  early  user 
familiarization  with  the  candidate  system,  test  facilities, 
and  test  trial  scenarios. 
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b.  Resource  Constraints  Category 

Shortcomings  associated  with  time,  personnel, 
equipment,  and  funding  availability  were  categorized  under 
Resource  Constraints.  These  shortcomings  impact  the  OT&E  test 
design  and  scope  in  terms  of  realism,  environmental  diversity, 
scale  of  threat  and  friendly  force  structures,  and  number  of 
test  event  iterations  completed.  The  degree  to  which 
resources  are  constrained  determines  the  operational  scope 
able  to  be  recreated  in  the  live  test,  and  the  type  and  amount 
of  data  available  for  evaluation. 

c.  System  Component   Stress  Category 

Stress  is  a  critical  element  of  the  operational 
environment  and  stress  level  reductions  on  personnel  or 
equipment  during  OT&E  testing  have  a  negative  impact  on  the 
evaluation.  Issues  resulting  from  unrealistic  equipment  and 
personnel  workloads,  threat  capabilities  encountered,  and 
familiarity  with  equipment  and  facilities  were  categorized 
under  System  Component  Stress.  OT&E  test  induced  stress 
levels  that  are  not  representative  of  the  operational 
environment  increase  the  risk  that  evaluations  are  based  on 
faulty  or  unrealistic  data. 

d.  Safety  and  Environmental  Concerns  Category 
Shortcomings  resulting  from  test  events  that  were 

eliminated  or  unrealistically  controlled  because  of  potential 
hazards  to  personnel  or  release  of  pollutants  into  the 
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environment  were  categorized  under  Safety  and  Environmental 
Concerns.  As  the  ranges,  lethality,  and  destructive  ability 
of  associated  systems  all  increase  with  new  technologies, 
safety  concerns  often  limit  the  use  of  the  actual  system  to 
very  controlled  circumstances  not  representative  of  a  combat 
environment.  Systems  that  create  radioactivity,  ozone 
depleting  elements,  and  other  environmentally  hazardous  by- 
products are  also  limited  to  controlled  scenarios. 
e.  Data  Validity  and  Reliability  Category 

Issues  associated  with  test  data  interpretation  or 
unusual  test  participant  qualifications  were  categorized  under 
Data  Validity  and  Reliability.  Live  test  events  contain  an 
infinite  number  of  variables,  providing  an  opportunity  for 
assigning  cause  to  undesirable  outcomes  and  eliminating 
outliers  or  biasing  data.  The  degree  to  which  test 
participants  represent  typical  operational  users  impacts  data 
validity  and  reliability. 

C.   ANALYSIS  OF  M&S  APPLICATION  TO  OT&E 

OT&E  problems  and  issues  within  each  of  the  identified 
categories  are  detailed  and  the  capabilities  of  M&S 
applications  to  resolve  them  are  analyzed  in  the  following 
paragraphs.  The  analysis  is  organized  by  problem  and  issue 
category. 
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1.   TEST  DESIGN 

The  OT&E  test  design  establishes  the  amount  of  data 
and  the  circumstances  in  which  data  are  collected.  The  test 
design  also  sets  the  foundation  for  the  amount  of  operational 
realism  achieved.  Creating  realistic,  combat  conditions  in 
OT&E  tests  have  been  hampered  by  many  factors .  Resource 
constraints,  discussed  in  the  following  section,  are  a  main 
cause  for  reduced  realism.  Accurate  TEMP  critical  issue 
interpretation  and  understanding  of  the  environments, 
scenarios,  and  forces  required  to  replicate  a  representative 
operational  environment  have  also  contributed  to  lack  of  test 
realism.  The  M1A2,  Air-to-Air  Target  Designator,  Javelin,  and 
Longbow  Apache  test  designs  had  significant  shortfalls. 
a.  M1A2   Main   Battle   Tank 

Interviews  with  TEC  personnel  indicated  that  the 
M1A2  test  was  limited  to  an  operational  microcosm  that 
significantly  increased  assessment  extrapolation  risks. 
Operational  deployment  of  the  M1A2  tank  at  the  brigade  or 
divisional  levels  could  not  be  assessed  due  to  the  resource 
constraints  which  limited  live  testing  to  one  battalion  sized 
task  force.  Battalions  deploy,  fight,  and  are  supported 
differently  than  brigades  or  divisions. 

A  constructive  simulation,  such  as  Janus,  appears 
to  have  application  in  addressing  operational  test  design 
shortfalls  similar  to  those  experienced  by  the  M1A2 .  Because 
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of  the  wide  range  of  force  structure,  environments,  threat 
forces,  and  combat  intensities  that  a  constructive  simulation 
is  capable  of  addressing,  it  is  useful  for  assessing 
operational  environments  and  scenarios  beyond  the  scope  of  a 
resource  constrained  live  test. 

TRAC  White  Sands  Missile  Range  (WSMR)  used  a 
similar  system  to  assist  in  the  M1A2  OT&E  test  scenario 
development.  Experiments  were  designed  on  a  Janus  based 
constructive  simulation  and  experiments  were  iteratively 
conducted  for  predictive  analysis.  Actual  test  equipment  were 
instrumented  and  data  were  collected  on  live  test  trials. 
Live  test  iterations  were  replicated  on  the  simulations  and 
then  correlated.  The  correlated  simulations  were  then  used  in 
other  environments  and  scenarios  within  the  M1A2  operational 
spectrum  and  analyzed.  The  insights  gained  from  these 
simulations  were  used  to  assist  in  the  M1A2  OT&E  test 
conducted  at  TEC.  OT&E  test  events  were  selected  based  on  the 
simulated  event  outcomes  that  indicated  a  high  probability  for 
obtaining  required  data  to  address  critical  issues. 
(Payan,1994) 

Jb.  Javelin  Missile 

TEC  interviews  revealed  that  battlefield  operating 
systems  normally  interoperating  with  Javelin  missile  systems 
were  not  included  in  the  Javelin  test  plan.  Javelin  is 
deployed   in   conjunction   with   other   anti-armor   systems 
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including  tanks,  artillery,  and  close  air  support.  This 
environment  was  too  expensive  and  impractical  to  replicate  in 
a  live  environment,  and  was  omitted  from  the  test  design. 

M&S  can  provide  a  method  for  addressing 
interoperability  issues  such  as  those  experienced  by  the 
Javelin  OT&E.  Constructive  simulations  are  capable  of 
representing  other  battlefield  operating  systems  and 
integrating  a  candidate  system's  capabilities  so  that 
interoperability  issues  can  be  assessed.  Virtual  simulations 
provide  methods  for  assessing  interoperability  issues  with 
personnel  operating  associated  battlefield  systems  within 
doctrinal  guidelines. 

Some  M&S  efforts  assisted  in  the  Javelin  OT&E  test 
design.  A  MTM  pre-test  model  conducted  on  the  Javelin  program 
successfully  correlated  live  OT&E  test  data  with  models 
generated  on  Janus.  Three  scenarios  with  six  missions  each 
were  conducted  on  the  model  including  offensive  and  defensive 
Javelin  employments.  These  resulting  data  provided  Javelin 
employment  assessments  in  an  operational  environment  and  OT&E 
test  scenarios  were  changed  to  incorporate  the  MTM 
recommendations.  Integrating  MTM  efforts  into  future  OT&E 
test  designs  can  improve  test  scenarios  and  environments,  and 
therefore,  enhance  operational  assessments.   (IR  MTM, 1993) 
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c.  Air-to-Air  Target  Designator 

Interviews  with  TEC  personnel  indicated  the  Air-to- 
Air  Target  Designator  OT&E  test  design  validation  process  was 
not  totally  effective.  The  test  design  validation  consisted 
of  a  paperwork  review  of  the  user  and  evaluator  data 
requirements  matched  against  the  test  design  plan.  A  pilot 
test  was  not  conducted  and  the  best  judgment  of  experienced, 
professional  OT&E  personnel  was  used  to  validate  the  test 
design  plan.  A  test  matrix  was  constructed,  like  the 
simplified  illustration  in  Table  1. 

TABLE  1   OPERATIONAL  TEST  MATRIX  EXAMPLE 


Test 
Events : 

Issues : 

Daylight  Live 
Fire 

Night  Live 
Fire 

96  Hour 
Field 
Training 
Exercise 

Can  the 

system  be 

effectively 

employed 

with  night 

vision 

equipment? 

X 

X 

Can  the 
system  be 
operated  in 
degraded 
modes? 

X 

X 

X 

Is  the 
system 
supportable 
by  user 
personnel? 

X 

X 

X 
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Operational  test  events  that  were  used  to  address  correlating 
critical  issues  are  designated  with  an  "X" .  The  matrix  was 
useful  in  ensuring  that  all  data  collection  requirements  were 
addressed  in  one  or  more  test  design  events.  However,  this 
paperwork  review  was  not  efficient  at  validating  the  scenarios 
and  environments  required  to  fully  address  the  issues. 

The  Air-to-Air  Target  Designator  test  design  plan 
validated  in  this  manner  had  serious  shortcomings  that  were 
solved  during  the  test  execution.  Night  operations  were 
specified  for  the  Air-to-Air  designator  OT&E.  Because  night 
operations  were  omitted  from  the  design,  a  major  test  event 
rescheduling  was  required  to  accommodate  the  night 
environments  specified.   (TEXC  Int,1994) 

An  OT&E  test  design  simulated  on  a  constructive 
simulation  appears  to  provide  a  method  for  validating  the  test 
design  plan.  The  critical  issues  could  be  input  into  the 
simulation  and  simulated  test  trials  conducted.  Scenarios 
could  then  be  adjusted  to  accommodate  all  critical  issues 
before  live  test  events  begin,  eliminating  shortfalls  such  as 
those  experienced  during  the  Air-to-Air  Target  Designator  OT&E 
test. 

M&S  was  used  in  the  Air-to-Air  Target  Designator 
OT&E  test  design.  The  program  used  constructive  simulations 
to  design  the  two-on-two  test  scenarios.  However,  no 
simulation  generated  data  were  used  for  the  report.  M&S  was 
used  to  assist  in  operational  test  scenario  development  and 
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test   scenarios   resulting   from   the   simulations   were 
successfully  implemented. 

d.    AH-64D  Longbow  Apache 

TEC  interviews  revealed  that  the  Longbow  Apache 
OT&E  test  design  had  threat  realism  shortfalls.  The  test 
design  did  not  provide  for  many  of  the  threat  assets  expected 
in  an  operational  environment.  Planned  threat  air  defense 
were  under-represented  and  lacked  realistic  electronic  jamming 
capabilities.  Threat  air  assets  were  under-represented  and 
excluded  from  many  planned  test  scenarios. 

M&S  could  provide  methods  for  addressing  threat 
realism  shortfalls  in  OT&E  test  designs.  Constructive 
simulations  can  replicate  the  full  range  of  threat 
capabilities  expected  in  an  operational  environment. 
Additionally,  virtual  simulations,  incorporating  SAFOR,  can 
provide  a  means  to  include  man-in-the-loop  threat  forces 
employed  by  human  beings  using  doctrinally  correct  threat 
tactics. 

M&S  was  used  in  the  Longbow  Apache  OT&E  test  design 
process.  TEXCOM  validated  the  scenarios  for  the  test  plan 
using  Janus  constructive  simulations.  In  addition,  the 
Longbow  Hellfire  missile  was  examined  in  various  electronic 
warfare  counter  measure  situations  using  a  hardware-in-the- 
loop  simulation.  These  pre-test  simulations  were  used  to 
establish  Hellfire  P„  and  PK  probabilities  for  casualty 
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assessments.  The  success  of  the  M&S  assisted  Longbow  Apache 
operational  testing  indicates  that  M&S  integration  can  improve 
the  test  design  process. 

e.  Test  Design  Summary 

In  virtually  all  OT&Es,  live  testing  represents 
only  a  small  operational  microcosm  for  the  candidate  system 
and  data  collected  are  therefore,  limited  to  that  microcosm. 
Live  operational  test  evaluations  are  limited  to  these  data 
collected.  Operational  assessments  beyond  the  test  scope  must 
be  extrapolated  to  address  the  system's  combat  effectiveness 
and  suitability.  As  the  scope  of  live  testing  is  reduced,  the 
evaluation  validity  to  the  operational  environments  not 
addressed  remains  unknown  and  risk  increases. 

M&S  integration  into  the  OT&E  test  design  appears 
to  provide  an  opportunity  for  addressing  areas  that  are  beyond 
the  scope  of  the  planned  testing.  M&S  applications,  including 
MTM,  constructive  simulations,  and  virtual  simulations  such  as 
the  Hellfire  hardware-in-the-loop,  can  enhance  the  OT&E  test 
design  process.  Candidate  systems  in  a  constructive  or 
integrated  simulation,  are  able  to  be  portrayed  at  every 
echelon  in  the  Army  structure,  under  varying  physical  and 
threat  environments,  and  interoperating  with  other  battlefield 
and  logistics  systems. 
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2.   RESOURCE  CONSTRAINTS 

Declining  DoD  budgets  increase  the  pressure  to  limit 
the  scope  of  testing  and  the  realism  that  can  be  included  in 
a  live  test.  Resource  Constraints  had  a  negative  impact  on 
the  M1A2,  Air-to-Air  Target  Designator,  Multi-Spectral  Combat 
Decoy,  and  Javelin  tests. 

a.    M1A2       Main       Battle       Tank,        Air-to-Air       Target 

Designator,    and  Javelin  Missile 

TEC  interviews  indicated  that  these  systems  were 
tested  at  the  small  unit  level,  so  the  interoperability  and 
impact  on  higher  echelon  units  could  not  be  ascertained  by  the 
OT&E  tests  conducted.  Similarly,  resource  constraints  limited 
the  inclusion  of  associated  battlefield  operating  systems  and 
support  elements  that  would  normally  be  deployed  in  a  combat 
environment.  Interoperability  issues  with  those  systems 
cannot  be  addressed  from  test  data  collected.  For  example, 
recovering  and  evacuating  an  M1A2  tank  from  the  battlefield  to 
the  corps  level  general  support  maintenance  unit  requires 
significant  recovery  and  transportation  assets  as  well  as 
multi-echelon  coordination.  M1A2  OT&E  test  resource 
constraints  did  not  allow  this  scenario  to  be  addressed  beyond 
the  participating  battalion's  organic  recovery  capabilities. 
(TEXC  Int. 1994) 

Interoperability  with  other  systems  beyond  the 
scope  of  live  testing  can  be  partially  assessed  through  M&S. 
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For  example,  the  TRAC  WSMR  simulations  addressed  M1A2 
capabilities  in  various  environments  and  scenarios  beyond  the 
scope  of  the  live  trials.  Using  similar  applications,  the 
M1A2  could  be  portrayed  interoperating  with  close  air  support, 
artillery  and  any  other  battlefield  operating  system  including 
critical  logistics  support  activities.  This  capability  is 
clearly  outside  of  the  live  test  environment's  resource 
constrained  limits. 

M&S  appears  to  have  application  in  addressing  OT&E 
test  shortfalls  created  by  resource  constraints.  The  marginal 
cost  of  M&S  generated  operational  events  is  extremely  low  as 
compared  to  those  same  events  conducted  in  the  live 
environment.  Resource  constraints  that  have  a  considerable 
impact  on  the  planning  of  live  events,  become  negligible  in 
the  M&S  environment. 

An  example  of  the  wide  disparity  between  live  and 
M&S  resource  requirements  involves  Ml  main  battle  tank 
prototypes.  In  1984,  evaluations  of  possible  improvements  on 
the  Ml  Abrams  tank  were  carried  out  by  using  real  tanks  in  a 
live  environment.  The  effort  took  24  months  and  cost  $40 
million.  A  later  effort  in  1986  used  a  modified  aircraft  dome 
simulator,  took  only  six  months  and  cost  $1  million.  In  1992, 
using  DIS,  four  variations  of  the  Ml  Abrams  were  operated 
against  potential  threats,  taking  only  three  months  and 
costing  $640,000.  The  use  of  M&S  can  reduce  resource 
constraint  impacts  by  providing  more  cost  effective  methods 
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for  assessing  candidate  systems'  operational  characteristics. 
(Berry/1992) 

Jb.  Multi-Spectral  Combat  Decoy 

Interviews  with  TEC  OT&E  test  officers  revealed 
that  resource  constraints  limited  the  scale  of  threat  forces 
planned  to  be  portrayed  during  Multi-Spectral  Combat  Decoy 
test  events.  The  system  was  not  subjected  to  the  full  range 
of  threat  detection  capabilities  that  would  be  expected  in  an 
operational  environment.  Threat  observation  systems  were 
limited  to  visual,  thermal,  and  radar  systems,  excluding  the 
extended  range  of  electro-optical  systems  available  to 
potential  threat  forces.   (TEXC  Int,1994) 

M&S  appears  to  provide  a  method  of  addressing 
threat  assets  beyond  the  scope  of  live  testing.  Using  the 
expected  detection  parameters  of  the  combat  decoys,  test 
trials  in  a  virtual  environment  provides  data  for  threat 
capabilities  that  cannot  be  represented  in  a  live  environment. 
However,  the  differences  in  observation  cross  sections  between 
actual  and  decoy  systems  must  be  quantifiable  and  verifiable. 
c.  Resource  Constraint   Summary 

Resource  limitations  impacted  realism  and  reduced 
the  scope  of  the  live  operational  test  performed.  Reduced 
realism  resulted  in  evaluations  that  were  based  on  data 
collected  from  test  trials  that  were  not  strictly 
representative  of  the  operational  system,  fielded  in  typical 
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Army  units,  in  a  combat  environment.  Reduced  scope  resulted 
in  less  quantitative  and  more  qualitative  evaluations  on  the 
candidate  systems . 

Virtual  and  constructive  simulations  can  enhance 
OT&E  tests  that  have  been  negatively  impacted  by  resource 
constraints.  Virtual  simulations  can  provide  methods  for 
representing  threat  and  friendly  force  capabilities  that  are 
too  expensive  or  impractical  to  replicate  in  a  live 
operational  test  environment.  Constructive  simulations  can 
provide  a  means  to  assess  portions  of  the  operational  spectrum 
outside  of  the  scope  of  the  live  test. 
3.   SYSTEM  COMPONENT  STRESS 

The  stress  imposed  on  personnel  and  equipment  by  an 
operational  combat  environment  cannot  be  replicated  through 
constructive,  virtual,  or  live  simulation.  Therefore,  live 
testing,  while  replicating  the  combat  environment,  is  the  most 
desirable,  but  most  expensive  substitute.  TEC  interviews 
revealed  that  personnel  and  equipment  components  were  not 
stressed  to  levels  expected  in  an  operational  environment  in 
the  Air-to-Air  Target  Designator,  Longbow  Apache,  and  MELIOS 
OT&Es. 

a.  Air-to-Air  Target    Designator    and    AH-64D    Longbow 

Apache 

The  Air-to-Air  Target  Designator  test  was  planned 
for  two-on-two  engagements,  but  the  operational  capability  of 
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the  designator  specified  force  ratios  of  up  to  two-on-six 
engagements.  Helicopter  crews  in  both  tests  had  limited  duty 
cycles,  not  representative  of  an  operational  combat 
environment.  In  addition,  the  same  ranges  and  maneuver  spaces 
used  for  operational  test  participant  training  were  used  for 
test  trials.  Familiarity  with  test  facilities  were  considered 
to  improve  reactions,  decisions,  and  weapons  accuracy.  As 
previously  discussed,  threat  capabilities  were  under- 
represented  which  also  reduced  personnel  stress  in  these 
tests.   (TEXC  Int,1994) 

M&S  provides  methods  for  enhancing  personnel  stress 
without  creating  dangerous  environments.  Virtual  simulations, 
integrating  operational  pilots  on  flight  simulators  with  SAFOR 
simulated  threat  aircraft,  would  provide  the  stress  associated 
with  multiple  aircraft  engagements.  Missions  could  also  be 
extended  to  combat  durations  using  simulations  without 
jeopardizing  crews  or  eguipment.  Data  resulting  would  reflect 
more  realistic  personnel  stress  levels  than  are  possible  in  a 
peace  time  environment. 

Training  test  participants  via  virtual  simulations 
preserves  personnel  stresses  induced  by  unfamiliar 
environments.  Training  could  also  be  accomplished  through 
distributed  simulations  using  actual  eguipment. 

Threat  induced  personnel  stress  is  enhanced  through 
M&S  integration.  M&S  applications  provide  the  full  range  of 
threat  capabilities.   Simulations  like  BDS-D  and  SAFOR,  have 
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the  requisite  threat  signatures  and  capabilities.  Test 
participants  operating  in  virtual  or  integrated  simulations 
can  be  opposed  by  the  full  threat  capacity,  controlled  by 
thinking  human  beings  in  realistic  environments.  These  threat 
forces  are  often  more  realistically  reproduced  in  M&S  than  is 
practical  in  a  live  environment.  Integrating  virtual 
simulations  enhances  the  stress  on  test  participants  to  more 
realistic  levels,  therefore,  operational  assessments  are  more 
accurate. 

b.  MELIOS 

MELIOS  experienced  limitations  in  the  degree  to 
which  personnel  were  realistically  stressed.  Duty  cycles  for 
the  MELIOS  test  trials  were  considerably  shorter  than  the 
specified  operational  cycle.  System  induced  fatigue  effects 
on  operators  was  not  determined. 

Conducting  trials  with  operational  users  through 
virtual  simulations  provides  a  means  to  extend  duty  cycles 
when  similar  live  testing  cannot  be  accommodated.  The  impact 
of  more  representative  fatigue  can  be  assessed  from  M&S 
generated  data.  The  risks  to  personnel  and  equipment  from 
fatigued  operators  is  minimized  by  conducting  extended  duty 
cycles  on  virtual  simulations  rather  than  on  actual  equipment. 

c.  System  Component  Stress  Summary 

The  stress  imposed  on  personnel  and  equipment  by  an 
operational  combat  environment  cannot  be  replicated  through 
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constructive,  virtual,  or  live  simulation.  Live  simulation  of 
the  combat  environment  is  the  most  desirable  and  most 
expensive  means  of  conducting  OT&E.  However,  the  live  test 
addresses  only  a  small  microcosm  of  the  operational 
environment  spectrum. 

M&S  integration  appears  to  provide  a  means  to 
preserve  some  personnel  stress  in  the  conduct  of  actual  live 
test  trials.  Virtual  simulations  using  SAFOR  can  provide 
representative  threat  capabilities  which  could  contribute  to 
threat  induced  personnel  stress.  Training  test  participants 
through  virtual  and  distributed  simulations  could  also 
preserve  some  personnel  stress  associated  with  unfamiliar 
environments . 

4.   SAFETY  AND  ENVIRONMENTAL  CONCERNS 

Managing  the  safety  and  environmental  impacts  of  new 
technology  testing,  negatively  impacts  on  the  successful 
creation  of  a  realistic  combat  environment  where  safety  and 
environmental  side  effects  are  not  so  important.  Safety  and 
environmental  issues  affected  the  M1A2,  Air-to-Air  Target 
Designator,  Javelin,  and  Longbow  Apache  OT&E  tests. 

a.    Air-to-Air    Target    Designator    and    AH-64D    Longbow 

Apache 

TEC  test  officers  revealed  that  peace  time  crew 
rest  requirements  for  helicopters  in  the  Air-to-Air  Target 
Designator  and  Longbow  Apache  tests  were  strictly  adhered  to. 
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These  crew  rest  requirements  are  not  representative  of  combat 
crew  duty  cycles,  which  are  not  constrained  to  peace  time 
training  restrictions.  As  previously  mentioned,  safety 
concerns  associated  with  maneuvering  more  than  four  aircraft 
in  close  airspace  limited  Air-to-Air  Target  Designator  trials 
to  two-on-two  engagements. 

M&S  applications  providing  safe  environments  could 
be  used  to  assess  the  impact  of  extended  operations  on 
aircraft  crews.  Safety  concerns  make  this  the  only  practical 
method  for  obtaining  this  type  of  operational  data. 

With  no  actual  threat  to  the  safety  of  soldiers, 
safety  considerations  are  nearly  non-existent  in  M&S 
applications.  For  example,  relatively  close  air-to-air  combat 
involving  more  than  the  safety  constrained  two-on-two  force 
ratios  are  probable  in  an  operational  environment.  Virtual  or 
constructive  simulations  provide  a  means  to  address  issues 
involving  more  than  two-on-two  force  ratios.  While  realism  is 
reduced,  safety  and  environmental  concerns  are  accommodated 
through  M&S  integration  without  the  total  loss  of  data. 

b.    M1A2  Main  Battle  Tank,  Javelin  Missile,   and  AH-64D 

Longbow  Apache 

TEC  interviews  indicated  that  live  fire  ranges 
limited  the  operational  employment  planning  for  the  M1A2, 
Javelin,  and  Longbow  Apache  systems.  The  ranges  used  were 
well-marked  for  firing  limits  and  oriented  the  crews  toward 
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target  positions.  Lasers,  in  all  but  the  Javelin  Missile, 
were  restricted  to  controlled  maneuver  and  air  spaces.  These 
controlled  maneuver  and  air  spaces  did  not  provide 
representative  maneuver  space  or  engagement  ranges. 
Environmental  concerns  eliminated  operational  firing  of 
depleted  uranium  M1A2  main  gun  rounds  due  to  radiation 
hazards.   (TEXC  Int,1994) 

Both    constructive    and   virtual    simulation 

applications  provide  environments  that  are  not  affected  by 

these  types  of  safety  and  environmental  considerations.  As  a 

result,  data  generated  on  M&S  are  not  biased  by  those  factors. 

c.  Safety  and  Environmental  Concerns  Summary 

Safety  and  environmental  concerns  negatively  impact 
on  the  ability  to  assess  the  operational  effectiveness  and 
suitability  of  candidate  systems.  Integrating  M&S  into  the 
OT&E  process  provides  a  means  for  assessing  issues  in  an 
environment  that  is  not  constrained  by  safety  and 
environmental  concerns.  Virtual  and  constructive  simulation 
applications  can  be  used  to  assess  issues  that  cannot  be 
safely  or  environmentally  conducted  in  a  live  operational 
test.  When  test  events  are  limited  in  number  due  to  safety  or 
environmental  concerns,  post-test  MTM  can  provide  a  method  to 
extend  data  and  enhance  operational  evaluations. 
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5.   TEST  DATA  VALIDITY  AND  RELIABILITY 

The  myriad  of  variables  present  in  a  live  test 
environment  impact  on  data  validity  and  reliability.  The 
M1A2,  Air-to-Air  Target  Designator,  Multi-Spectral  Combat 
Decoy,  and  MELIOS  tests  experienced  data  validity  and 
reliability  problems. 

a.  Multi-Spectral   Combat  Decoy  and  MELIOS 

TEC  interviews  indicated  that  both  the  Multi- 
Spectral  Combat  Decoy  and  the  MELIOS  OT&E  tests  had  some 
unfavorable  test  event  outcomes  that  were  challenged  by 
parties  with  vested  interests  in  the  OT&E  test  results. 
Specific  events  were  subsequently  omitted  or  obscured  so  that 
test  trials  reflected  more  favorable  results  than  had  actually 
occurred.  For  example,  one  decoy  was  identified  (as  a  decoy) 
by  the  threat  participants  because  of  its  location  in  the 
maneuver  area.  A  protest  was  made  based  on  the  threat 
participant's  knowledge  of  the  maneuver  area  and  the  supposed 
errant  placement  of  the  decoy.  The  test  trial  was  eliminated 
from  the  test  report.  A  MELIOS  malfunction  was  traced  to  an 
operator  error.  The  finding  was  challenged  because  operator 
training  was  conducted  which  should  have  prevented  the 
malfunction.   Those  data  were  not  used  in  the  report. 

The  uniform  quality  of  M&S  generated  data  appears 
to  enhance  the  reliability  and  validity  of  OT&E  data.  M&S 
generated  data  are  less  susceptible  to  interpretation  as  the 


62 


M&S  application  controls  all  variables  by  design.  Fully 
verified,  validated  and  accredited  M&S  applications 
specifically  designed  for  data  collection  produce  uniform 
quality  data  which  should  enhance  the  evaluation  process. 

b.    M1A2     Main     Battle     Tank     and     Air-to-Air     Target 

Designator 

OT&E  personnel  at  TEC  indicated  that  the  M1A2  and 
Air-to-Air  Target  Designator  tests  had  problems  associated 
with  atypical  test  participants.  The  OT&E  test  participants 
were  tasked  from  tactical  units  for  the  test  duration.  As 
they  were  "representatives"  of  the  tactical  unit  tasked  with 
support,  many  were  higher  quality  in  terms  of  training, 
aptitude,  experience,  and  attitude  than  typical  operational 
individuals.  Conducting  the  tests  with  these  "golden  crews" 
created  some  degree  of  test  data  bias.  For  example,  the 
pilots  in  the  Air-to-Air  Target  Designator  test  were  all 
senior  chief  warrant  officers  (CW3  or  CW4 )  with  thousands  of 
flying  hours  logged.  Their  pilot  skills  were  considered  to  be 
atypically  high  when  compared  to  less  experienced  pilots 
prevalent  in  the  operational  environment. 

M1A2  test  participants  were  also  considered  to  be 
atypically  qualified.  Training  and  familiarization  of  the 
M1A2  test  crews  were  conducted  on  the  same  ranges  and  maneuver 
areas  as  the  OT&E  test  trials.  Decisions,  response  times,  and 
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accuracy  of  weapons  firing  were  considered  superior  to  what 
could  be  expected  in  a  combat  environment.   (TEXC  Int,1994) 

The  distributive  nature  of  DIS  provides  an 
opportunity  to  address  the  "golden  crew"  problem.  Test 
participants  selected  over  a  wide  geographic  area  and  linked 
together  through  DIS,  can  participate  without  leaving  their 
home  station.  The  tendency  to  select  atypical  users  at  home 
station  would  be  reduced  because  participants  remain  anonymous 
and  are  not  representing  the  unit  as  individuals.  The  same 
techniques  could  be  used  to  train  test  participants  without 
familiarizing  them  with  the  limited  test  ranges  and  maneuver 
areas  used  for  actual  test  trials.  Data  collected  in  this 
manner  would  have  enhanced  validity  and  reliability,  thus 
improving  the  evaluation. 

c.    Data  Validity  and  Reliability  Summary 

Data  validity  and  reliability  are  affected  by  the 
OT&E  test  scope,  realism  achieved,  and  the  degree  to  which 
test  participants  represent  typical  users.  Operational 
assessments  of  combat  effectiveness  and  suitability  are 
directly  affected  by  data  validity  and  reliability. 
Integrating  M&S  into  the  OT&E  process  through  the  use  of  MTM, 
virtual,  constructive,  and  distributed  simulations,  enhances 
data  reliability  and  validity,  and  therefore,  assessments 
become  more  accurate. 
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D.   SUMMARY 

Integration  of  advanced  M&S  techniques,  technologies,  and 
applications  including  Model-Test-Model,  Janus,  BDS-D,  SAFOR, 
and  DIS,  appears  to  provide  a  means  of  addressing  operational 
issues  beyond  the  scope  of  resource  constrained  live  testing. 
M&S  appears  to  provide  a  vehicle  for  addressing  critical 
issues  that  are  impossible  or  impractical  to  represent  in  a 
live  environment.  Operational  realism  and  system  stress 
appear  to  be  enhanced  through  M&S  application. 

Integrating  M&S  does  not  replace  or  eliminate  live 
operational  testing,  but  rather  augments  and  focuses  live 
testing  where  results  would  provide  the  best  data  for  the 
operational  evaluation.  The  adage  "you  don't  use  M&S  to 
identify  what  is  exactly  right,  you  use  it  to  eliminate  what 
is  exactly  wrong"  applies  to  OT&E  planning.  Integrating  M&S 
into  OT&E  planning  reduces  variability  by  examining  critical 
issues  across  the  operational  spectrum  replicated  on  a 
simulation. 
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V.   CONCLUSIONS  AND  RECOMMENDATIONS 

A/   INTRODUCTION 

This  thesis  focused  on  opportunities  to  integrate  M&S 
concepts  and  applications  into  the  Army  OT&E  process. 
Recurring  OT&E  problems  and  issues  were  examined  and  possible 
M&S  based  enhancements  and  solutions  were  identified.  Army 
OT&E  process  improvements  through  M&S  integration  were 
analyzed. 

The  OT&E  process  has  a  major  impact  on  candidate  systems' 
critical  acquisition  decisions.  From  early  operational 
assessments  through  the  formal  operational  evaluation,  the 
insights  to  the  systems'  operational  characteristics  are  based 
on  OT&Es.  The  accuracy  of  these  OT&Es  in  predicting  how 
systems  will  operate  in  a  combat  environment  directly  impacts 
on  future  mission  accomplishment,  and  equally  important,  the 
survival  of  the  combat  troops  who  depend  upon  those  systems. 

The  importance  of  OT&Es  in  the  acquisition  process  is 
underscored  by  the  Congressional  interest  in  the  OT&E  process 
and  results.  OT&E  assessments  are  reported  directly  to 
Congress  from  independent  OT&E  sources  through  the  DOTE. 
Continued  program  authorization  and  funding  decisions  are 
influenced  by  the  operational  information  provided. 
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M&S  integration  into  the  OT&E  process  has  traditionally 
been  limited.  In  the  past,  M&S  maturity  was  limited  to 
physical  models  and  mathematical  simulations.  Advancements  in 
supporting  areas  from  software  to  communications  networks  have 
resulted  in  explosive  M&S  growth  and  maturity.  However,  the 
OT&E  community  has  been  slow  to  integrate  this  advanced  M&S 
due  to  the  desire  to  protect  their  independent  status  and  the 
perceived  negative  connotation  associated  with  anything 
"simulated"  in  a  testing  environment. 

B.   GENERAL  CONCLUSIONS 

Operational  combat  effects  on  personnel  and  equipment 
cannot  be  wholly  replicated  through  constructive  simulation, 
virtual  simulation,  or  live  operational  testing.  Live  testing 
of  the  combat  environment  provides  the  closest  possible 
substitute.  It  is  the  most  desirable,  but  most  expensive 
method  for  addressing  OT&E  critical  issues.  M&S  cannot,  and 
should  not  be  used  in  place  of  necessary  OT&E  live  testing. 
However,  M&S  applications  offer  methods  of  augmenting  and 
enhancing  the  OT&E  process  that  merit  consideration. 

The  requirement  to  create  holistic,  representative 
operational  environments  conflicts  with  the  realities  of 
reduced  current  and  projected  defense  funding  authorizations. 
The  tremendous  costs  associated  with  conducting  realistic, 
comprehensive  OT&Es  makes  them  susceptible  to  reduced  funding 
pressures.  Army  budgets  are  projected  to  continue  the  present 
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downward  trend,  which  will  not  improve  the  prospects  for 
expanding  OT&E  live  testing. 

Interviews  and  other  investigations  indicate  that  there 
ate  significant  problems  and  issues  associated  with  the  Army 
OT&E  process.  These  recurring  problems  and  issues  negatively 
impact  on  OT&E  abilities  to  adequately  assess  the  combat 
effectiveness  and  suitability  of  candidate  systems,  as 
required  by  law.  Because  of  the  identified  problems  and 
issues,  OT&E  tends  to  yield  more  favorable  assessments  than 
are  likely  to  be  found  when  the  systems  are  employed  in 
combat.  This  can  lead  to  the  funding  of  weapon  systems  whose 
operational  effectiveness  and  suitability  have  not  been 
demonstrated. 

The  integration  of  M&S  including  MTM,  constructive 
simulations,  SAFOR,  virtual  simulations,  and  distributed 
simulations  can  provide  some  solutions  to  recurring  OT&E 
problems  and  issues.  M&S  cannot  fully  represent  the  personnel 
stresses  associated  with  combat  environments.  M&S  cannot 
replicate  the  infinite  number  of  variables  present  in  live 
environments.  However,  integrating  M&S  into  the  OT&E  process 
can  provide  valuable  insights  into  the  operational  environment 
beyond  the  scope  of  a  resource  constrained,  live  operational 
test. 
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C.   SPECIFIC  CONCLUSIONS 

Specific   Conclusions   regarding   M&S   application   in 
addressing  OT&E  recurring  problems  and  issues  are  presented  in 
the  following  paragraphs. 
1.   Test  Design 

a.  Current  Problems  and  Issues 

Interviews  with  OT&E  professionals  at  TEC  indicated 
the  following  OT&E  test  design  recurring  problems  and  issues: 


•  It  is  generally  impossible  or  impractical  to  portray  the 
full  range  of  force  structures,  interoperating  systems, 
and  potential  threat  capabilities  faced  by  even  the  most 
basic  system. 

•  The  current  methods  for  planning  the  OT&E  test,  and 
validating  that  plan  are  not  efficient  or  effective. 

b.    M&S  Solutions 

Integrating  M&S  into  the  OT&E  test  design  process 
can  assist  planners  in  addressing  the  problems  and  issues, 
specifically: 

•  Constructive  simulations  can  provide  a  means  of  examining 
the  candidate  system's  entire  operational  spectrum. 
Force  structures,   interoperating  systems,   and  threat 
capabilities  can  be  represented  in  the  constructive 
environment . 

•  Integrating  planned  OT&E  live  test  events  with 
constructive  simulations  and  Model-Test-Model  applications 
can  identify  scenarios  and  environments  required  to 
address  critical  issues. 
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2.  Resource  Constraints 

a.  Current  Problems  and  Issues 

TEC  interviews  revealed  the  following  recurring 
problems  and  issues  associated  with  resource  constraints: 

•  Resource  constraints  limit  the  scope  of  OT&E  live  testing. 

•  The  operational  microcosm  considered  shrinks  as  resources 
are  constrained.  This  increases  the  risk  that  the  OT&E 
will  neither  accurately  nor  adequately  address  the  combat 
operational  environment  that  potential  weapon  systems 
might  experience. 

•  Resource  constraints  limit  the  degree  to  which  realism  is 
recreated  in  a  live  environment. 

b.  M&S  Solutions 

Integrating  M&S  into  OT&E  testing  can  reduce  the 
impact  created  by  resource  constraints : 

•  Constructive  and  virtual  simulations  can  provide  methods 
for  OT&E  to  address  operational  issues  beyond  the  scope  of 
resource  constrained  live  testing. 

•  M&S  applications  are  more  cost  effective  than  live  test 
events  and  therefore,  are  less  susceptible  to  the  negative 
effects  resulting  from  resource  constraints. 

•  Virtual  simulations  accommodate  actual  users  in  an 
unconstrained  environment.  This  provides  critical  user 
responses  in  environments  with  realistic  threat 
capabilities,  interoperability  with  other  battlefield 
operating  systems,  and  force  structures  impossible  to 
replicate  in  a  live  environment. 

3.  System  Component  Stress 

a.    Current  Problems  and  Issues 

OT&E  test  personnel  at  TEC  identified  the  following 
problems  and  issues  related  to  system  component  stress: 
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•  Stress  on  OT&E  test  participants  and  equipment  is  usually 
not  commensurate  with  levels  expected  in  an  operational 
environment . 

•  Resource  constraints  that  limit  threat  capabilities, 
workload,  and  required  interoperability  functions,  may 
reduce  6tress  to  unrealistic  levels  in  the  live 
environment. 

b.    M&S  Solutions 

M&S  integration  can  preserve  or  restore  more 
realistic  personnel  stress  levels  to  OT&E  test  participants, 
specifically: 

•virtual  and  distributed  simulations,  used  as  a  training 
vehicle  for  test  participants,  can  preserve  the  stresses 
associated  with  unfamiliar  weapons  engagement  ranges  and 
maneuver  areas  used  exclusively  for  live  test  trials. 

•  Threat  forces  represented  on  virtual  and  constructive 
simulations,  can  restore  the  stress  of  confronting  test 
participants  with  the  full  range  and  capability  of 
representative  threat  forces,  operated  by  threat  expert 
personnel . 

•  Stimulators  can  provide  equipment  stress  for  software 
intensive  and  communications  systems.  Other  forms  of 
equipment  related  physical  stress  cannot  be  reproduced 
through  M&S  applications. 

4.   Safety  and  Environmental 

a.  Current   Problems  and  Issues 

TEC  interviews  identified  the  following  safety  and 

environmental  issues  that  increasingly  limit  the  ability  to 

fully  consider  OT&E  critical  issues  in  a  live  environment: 

•New  technology  advancements  including  lasers,  depleted 
uranium  projectiles,  and  other  potentially  hazardous 
systems  cannot  be  addressed  in  an  unrestricted,  live 
operational  test. 
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•  Systems  that  produce  environmentally  hazardous  by-products 
are  restricted  to  very  controlled  scenarios,  not 
representative  of  a  combat  environment. 

•  Peace  time  safety  requirements  that  are  significantly  more 
restrictive  than  those  expected  in  a  combat  environment, 
negatively  impact  OT&E  assessments. 

b.    M&S  Solutions 

Integrating  M&S  into  the  OT&E  process  can  provide 
solutions  for  data  collection  when  scenarios  are  restricted  by 
safety  or  environmental  concerns: 

•  M&S  can  provide  a  means  to  address  critical  issues  that 
are  too  hazardous  or  environmentally  polluting  to  conduct 
in  a  live  test.  In  extreme  situations,  M&S  may  provide 
the  only  OT&E  data  obtainable  for  OT&E  consideration. 

5.   Data  Validity  and  Reliability 

a.  Current  Problems  and  Issues 

Interviews  with  TEC  OT&E  personnel  revealed  the 
following  data  validity  and  reliability  problems  and  issues: 

•  Data  collected  from  the  OT&E  microcosm  must  be 
extrapolated  to  the  operational  spectrum  not  addressed 
within  the  live  OT&E  scope.  As  OT&E  test  scope  becomes 
smaller  and  more  constrained,  data  validity  and 
reliability  risks  increase. 

•  The  myriad  of  variables  present  in  a  live  OT&E  environment 
invites  questionable  interpretation  of  the  results.  False 
causes  may  be  assigned  to  some  undesirable  events  that 
eliminate,  obscure,  or  reduce  the  severity  of  the  failure. 

•  OT&E  test  participants  are  often  not  representative  of  the 
typical  operational  user.  Data  validity  and  reliability 
suffer  when  "golden  crews"  (personnel  who  are  atypically 
high  in  aptitude  and  attitude)  are  used  to  represent  the 
"typical"  user. 

•  Training  test  participants  on  the  same  ranges  and  maneuver 
areas  used  for  OT&E  test  trials  creates  unrealistic 
responses.  This  familiarity  improves  response  times, 
anticipation,  and  weapons  accuracy. 
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b.    M&S  Solutions 

Integrating  M&S  into  the  OT&E  process  can  assist  in 
addressing  data  validity  and  reliability  related  problems  and 
issues: 

•  M&S  applications  can  provide  a  means  to  augment  data 
required  for  quantitative  evaluation. 

•  Data  generated  on  M&S  applications  have  controlled 
variables,  virtually  eliminating  the  opportunity  for  event 
interpretation  and  manipulation. 

•  DIS  can  provide  an  effective  means  for  addressing  the 
"golden  crew"  problem.  Data  collection  and  test 
participant  training  via  DIS  preserve  the  actions, 
reactions,  and  decisions  of  representative  operational 
users . 

D.   RECOMMENDATIONS 

Implementing  the  following  recommendations  should  enable 
M&S  to  enhance  and  improve  the  Army  OT&E  process. 

1.  The  Army  should  continue  and  expand  existing  M&S 
efforts  in  requirements  generation,  data  augmentation,  and 
OT&E. 

Cooperative  M&S  efforts  between  OPTEC  and  TRADOC,  such 
as  the  M1A2  and  Javelin  programs,  will  enhance  the  OT&E 
process  through  early  and  accurate  critical  issue 
identification  and  test  design  validation.  OT&E  test  data 
will  be  improved  and  combat  effectiveness  and  suitability 
assessments  will  be  more  accurate. 
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2.  M&S  applications  that  have  the  fidelity  to  integrate 
candidate  system  capabilities  and  parameters  for  OT&E  should 
be  developed. 

The  M&S  applications  developed  should  portray  existing 
threat  and  friendly  force  structures  in  varying  environments 
through  theater  level.  These  applications  should  include 
constructive  and  virtual  simulations  capable  of  examining  the 
impacts  on  threat  capability,  battlefield  interoperability, 
and  logistics  supportability  issues.  This  will  provide 
methods  for  examining  the  operational  spectrum  beyond  the  live 
test  scope.  Extrapolation  of  live  test  data  via  M&S  will 
enhance  data  and  evaluation  accuracy. 

3.  Verification,  Validation  and  Accreditation  (W&A) 
requirements  for  previously  validated  applications  should  be 
limited  to  the  candidate  systems '  capabilities  and  parameters 
added . 

The  W&A  process  for  M&S  applications  is  time 
consuming  and  expensive.  M&S  applications  with  standing  W&A 
approval  could  enhance  M&S  integration  into  the  OT&E  process. 

4.  Equipment  simulators  at  user  locations  should  be 
linked  through  DIS  for  virtual  simulations  integrating  a  wide 
variety  of  typical  users  for  operational  assessments. 

Test  participant  training  at  home  station  through  DIS 
could  improve  OT&E  data  validity  during  live  test  trials. 
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E.   FURTHER  RESEARCH 

Further  research  is  recommended  on  the  following  subjects 
relating  to  this  thesis: 

1.  Research  should  be  conducted  to  determine  the 
differences  between  test  participant  actions,  reactions,  and 
decisions  in  live  versus  simulated  environments. 

The  results  of  this  research  should  help  establish  the 
degree  to  which  M&S  generated  reaction  and  decision  data 
correlate  with  live  operational  testing. 

2 .  Research  should  be  conducted  on  the  impacts  of  the 
future  digitized  battlefield  on  live  and  simulated  OT&E  test 
events . 

The  effects  of  the  future  digitized  battlefield  on 
tactics  and  user  decisions  must  be  replicated  in  OT&E  and 
related  M&S  applications.  This  will  be  required  to  accurately 
portray  the  operational  environment  under  battlefield 
digitization  in  future  M&S  applications. 

3.  The  compatibility  of  current  and  future  weapon  system 
training  simulation  devices  with  DIS  and  other  M&S 
architectures  should  be  investigated. 

Dual  use  of  training  simulation  devices  for  training 
and  OT&E  testing  via  virtual  and  distributed  simulations  could 
enhance  the  M&S  integration  into  the  OT&E  process. 

4.  The  integration  of  artificial  intelligence  with 
operationally  related  M&S  applications  should  be  analyzed. 
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Candidate  system  employment  via  artificial 
intelligence  simulations  could  provide  effectiveness  and 
suitability  assessments  while  varying  doctrine  and  tactics. 
Threat  forces  controlled  through  artificial  intelligence 
simulations  could  provide  probable  threat  adjustments  to  the 
employment  of  candidate  weapon  systems . 
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APPENDIX 


Acq Acquisition 

ADATS-A  Acq  and  Dev  of  Threat  Simulators  Activity 

AFOTEC   .   Air  Force  Operational  Test  and  Evaluation  Command 

AMC Army  Materiel  Command 

AR Army  Regulation 

ARPA   Advanced  Research  Projects  Agency 

ASA Assistant  Secretary  of  the  Army 

ASA(RDSA)  .  .  .  ASA  (Research,  Development,  and  Acquisition) 

ASARC  Army  Systems  Acquisition  Review  Council 

ATD  Advanced  Technology  Demonstration 

BDS-D  .  .  Battlefield  Distributed  Simulation  -  Developmental 
BLG  Battle  Laboratory  Guide 

C4I  .   Cmd,  Control,  Communications,  Computers,  Intelligence 

CEP  Concept  Evaluation  Program 

COEA   Cost  and  Operational  Effectiveness  Analysis 

DAB  Defense  Acquisition  Board 

DIS  Distributed  Interactive  Simulation 

DMSO   Defense  Modeling  and  Simulation  Office 

DoD  Department  of  Defense 

DoDI DoD  Instruction 

DOTE   Director  of  Operational  Test  and  Evaluation 

DSI  Defense  Simulation  Internet 

DT&E   Developmental  Test  and  Evaluation 

DUSA Deputy  Under  Secretary  of  the  Army 

DUSA(OR)   DUSA  (Operations  Research) 

ECM    Electronic  Counter  Measures 

EUTE Early  User  Test  and  Evaluation 

FDTE   ....   Force  Development  Testing  and  Experimentation 

FOTE  Follow-on  Operational  Test  and  Evaluation 

FY Fiscal  Year 

IOTE   Initial  Operational  Test  and  Evaluation 

1ST  Institute  for  Simulation  and  Training 

JT&E Joint  Test  and  Evaluation 

LAM  Louisiana  Maneuvers 

LRIP Low  Rate  Initial  Production 
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LTG  Lieutenant  General 

M&S  Modeling  and  Simulation 

MELIOS   .  .  .  Mini  Eyesafe  Laser  Infrared  Observation  System 

MS Milestone 

MTM  Model-Test-Model 

OEC  Operational  Evaluation  Command 

OPEVAL   Operational  Evaluation 

OPTEC  Operational  Test  and  Evaluation  Command 

OR   Operations  Research 

ORD  Operational  Requirements  Document 

OSD Office  of  the  Secretary  of  Defense 

OT Operational  Test 

OTSA   Operational  Threat  Support  Agency 

OT&E Operational  Test  and  Evaluation 

OTEA   Operational  Test  and  Evaluation  Agency 

PH   Probability  of  Hit 

PK   Probability  of  Kill 

PDU Protocol  Data  Unit 

R&D  Research  and  Development 

RD&A   Research,  Development  and  Acquisition 

RDT&E  Research,  Development,  Test  &  Evaluation 

S&T  Science  and  Technology 

SAFOR  Semi-Automated  Forces 

SIMNET   Simulation  Network 

STRICOM  .  .  Simulation,  Training  and  Instrumentation  Command 

T&E Test  and  Evaluation 

TEC  TEXCOM  Experimentation  Center 

TECO   Test  and  Evaluation  Coordination  Office 

TEMP Test  and  Evaluation  Master  Plan 

TEXCOM   Test  and  Experimentation  Command 

TMDE   Test,  Measurement,  and  Diagnostic  Equipment 

TRAC   TRADOC  Analysis  Command 

TRADOC   Training  and  Doctrine  Command 

USAPS  United  States  Army  Policy  Statement 

USC United  States  Code 

W&A   Verification,  Validation,  and  Accreditation 

VMS  Virtual  Memory  System 


78 


LIST  OF  REFERENCES 

Berry,  Clifton  F.,  "War  Breaker",  National  Defense,  December 
1992. 

Crooks,  William  H.,  and  Fraser,  Robert  E.,  "Design  Data 
Handbook  -  SIMNET/Janus  Interconnection",  IEI  Technical  Report 
No.  TR-17102-13000-1-05-92,  28  May  1992. 

Crouch,  Thomas  W. ,  "The  Impact  of  Modeling  and  Simulation  on 
the  Army  Acquisition  Process",  Naval  Postgraduate  School 
Thesis,  March  1994. 

Defense  Systems  Management  College,  "Test  and  Evaluation 
Management  Guide",  March  1988. 

Department  of  Defense  Instruction  5000.2,  "Defense  Acquisition 
Management  Policies  and  Procedures",  23  February  1991. 

Forster,  William  H.,  "Simulation  Support  to  Army  Acquisition", 
Memorandum  to  the  Deputy  Commander  Army  Materiel  Command  and 
Program  Executive  Officers,  24  May  1993. 

Holzer,  Robert,  and  Muradian,  Vago,  "U.S.  Army  Speeds 
Simulator  Program",  Defense  News,  7-13  June  1993. 

Institute  for  Simulation  and  Training,  "DIS  User's  Orientation 
Guide",  May  1993. 

Kelly,  Valerie,  "Topographic  Engineering  Center  Supports 
Simulation  and  Training",  Army  Research,  Development,  and 
Acquisition  Bulletin,  January-February  1993. 

LeSueur,  Stephen  C.,  "Senators:  DoD  Reform  Undermines  Weapon 
Test  Office",  Defense  News,  10-16  January  1994 

Loral  Systems  Company,  "A  Guidebook  for  Agencies  Conducting 
Research   Using   Battlefield   Distributed   Simulation 
Developmental  (BDS-D)",  5  May  1992. 

Maze,  Rick,  "Fallout:  Weapons  that  Don't  Work",  Army  Times, 
4  April  1994. 

Operational  Test  and  Evaluation  Command  Handbook  73-21-1, 
"Pocket  Guide  to  Modeling  and  Simulation  to  Support 
Operational  Test  and  Evaluation",  9  December  1993. 


79 


Pate,  Charles  A.,  and  Proctor,  Michael  D.,  "Interim  Report  on 
Model-Test-Model  in  Support  of  the  Javelin  IOTE",  December 
1993. 

Pate,  Maria  C,  "Comparison  of  Janus  and  Field  Test  Helicopter 
Engagement  Ranges  for  the  Line-of-Sight  Forward  Heavy  System", 
Naval  Postgraduate  School  Thesis,  December  1992. 

Payan,  Ferdie,  "Telephone  Interview",  White  Sands  Missile 
Range,  15  August  1994. 

Ross,  Jimmy  D.,  "Legacy  for  the  '90's  in  Louisiana  Maneuvers", 
Army  Magazine,  June  199  3. 

Singley,  George  T.  Ill,  "Distributed  Interactive  Simulation  - 
A  Preview",  Army  Research,  Development,  and  Acquisition 
Bulletin,  March-April  1993. 

TEXCOM  Experimentation  Center,  Combat  Experimentation  Branch, 
"Personal  Interviews",  June  1994. 

Trifiletti,  Anthony  C,  "Personal  Interview",  Naval 
Postgraduate  School,  10  February  1994. 

United  States  Army  Materiel  Command  Public  Affairs  Office, 
"How  to  Do  Business  with  Battle  Labs:  A  Guide  for  Industry", 
May  1993. 

United  States  Army  Regulation  73-1  (Draft),  "Test  and 
Evaluation",  Undated. 

United  States  Code,  Title  10,  1989. 

West,  Togo  D.  Jr.,  and  Sullivan,  Gordon  R.  ,  "Challenges  and 
Opportunities",  United  States  Army  Posture  Statement  FY  95, 
February  1995. 

Williams,  Robert  H.,  "Simulation  Provides  Key  to  Drawdown 
Dilemma",  National  Defense,  May- June  1993. 


80 


INITIAL  DISTRIBUTION  LIST 


No.  Copies 

1.  Defense  Technical  Information  Center  2 
Cameron  Station 

Alexandria  VA  22304-6145 

2.  Library,  Code  052  2 
Naval  Postgraduate  School 

Monterey  CA  93943-5002 

3.  Defense  Logistics  Studies  Information  Exchange         1 
U.S.  Army  Logistics  Management  Center 

Fort  Lee  VA  23801-6043 

4.  OASA  (RDA)  1 
ATTN:   SARD-ZAC 

103  Army  Pentagon 
Washington  DC  20310-0103 

5.  Professor  David  V.  Lamm,  Code  SM/Lt  5 
Department  of  Systems  Management 

Naval  Postgraduate  School 
Monterey  CA  93943-5000 

6.  Lieutenant  Colonel  Brad  R.  Naegle  2 
304  S.  500  E. 

Clearfield  UT  84015 

7.  Professor  Thomas  H.  Hoivik,  Code  SM/Ho  1 
Department  of  Systems  Management 

Naval  Postgraduate  School 
Monterey,  CA  93943-5002 


81 


.   ■ 


r 

■ 


GAYLORD  S 


